2024-04-25T06:37:12.915 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T06:37:12.915 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672895 --name nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi --owner scheduled_nmordech@teuthology --verbose --description rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} -- /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672895/orig.config.yaml 2024-04-25T06:37:12.937 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T06:37:13.016 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672895 branch: main description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: null first_in_suite: false job_id: '7672895' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f workunit: branch: wip-nitzan-osd-recovery-standalone-test-wait-for-too-full sha1: a08304379f921e5e229e92c75ede8800059313ff owner: scheduled_nmordech@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7777 sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f sleep_before_teardown: 0 suite: rados:standalone suite_branch: wip-nitzan-osd-recovery-standalone-test-wait-for-too-full suite_path: /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa suite_relpath: qa suite_repo: https://github.com/NitzanMordhai/ceph.git suite_sha1: a08304379f921e5e229e92c75ede8800059313ff targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP7pb7rew8s6BVWAm6aasSxQlnZIH6WLJBLKbpCeqsG4Yz2raUgcvwq/SCfptdQVT2gItaGOnA3um+qCRPfT5E4= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_06:28:28 tube: smithi user: nmordech verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T06:37:13.016 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa; will attempt to use it 2024-04-25T06:37:13.016 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa/tasks 2024-04-25T06:37:13.017 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T06:37:13.018 INFO:teuthology.task.internal:Checking packages... 2024-04-25T06:37:13.044 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f' 2024-04-25T06:37:13.044 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T06:37:13.044 INFO:teuthology.packaging:ref: None 2024-04-25T06:37:13.044 INFO:teuthology.packaging:tag: None 2024-04-25T06:37:13.044 INFO:teuthology.packaging:branch: main 2024-04-25T06:37:13.044 INFO:teuthology.packaging:sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T06:37:13.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-04-25T06:37:13.280 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3253-g4e213a58-1jammy 2024-04-25T06:37:13.281 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T06:37:13.288 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T06:37:13.288 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T06:37:13.305 INFO:teuthology.task.internal:Saving configuration 2024-04-25T06:37:13.315 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T06:37:13.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T06:37:13.347 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672895', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 06:31:04.835480', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMPweClabYalOIb8iAnSyitaVJVb5hkvXl/rnE3n4M2'} 2024-04-25T06:37:13.347 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T06:37:13.353 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-25T06:37:13.353 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T06:37:13.391 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f26e1a293a0>, signals=[15]) 2024-04-25T06:37:13.391 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T06:37:13.397 INFO:teuthology.task.internal:Opening connections... 2024-04-25T06:37:13.397 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-04-25T06:37:13.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:37:13.478 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T06:37:13.485 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-04-25T06:37:13.587 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-04-25T06:37:13.587 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-04-25T06:37:13.631 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Ubuntu" 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="22.04" 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_CODENAME=jammy 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:ID=ubuntu 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE=debian 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T06:37:13.632 INFO:teuthology.orchestra.run.smithi088.stdout:UBUNTU_CODENAME=jammy 2024-04-25T06:37:13.633 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-04-25T06:37:13.654 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T06:37:13.662 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T06:37:13.667 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T06:37:13.667 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-04-25T06:37:13.679 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T06:37:13.685 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T06:37:13.685 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-04-25T06:37:13.731 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T06:37:13.763 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T06:37:13.777 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T06:37:13.777 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T06:37:13.777 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T06:37:13.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:37:13.782 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-04-25T06:37:13.828 INFO:teuthology.orchestra.run.smithi088.stdout:5.15.0-56-generic 2024-04-25T06:37:13.828 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.15.0-56-generic 2024-04-25T06:37:13.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get clean 2024-04-25T06:37:14.074 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2024-04-25T06:37:14.370 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:37:14.371 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T06:37:14.417 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T06:37:14.781 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T06:37:14.783 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T06:37:14.920 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [619 kB] 2024-04-25T06:37:15.086 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T06:37:15.123 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T06:37:15.259 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T06:37:15.271 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T06:37:15.271 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T06:37:15.274 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T06:37:15.289 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T06:37:15.290 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T06:37:15.353 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T06:37:15.354 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T06:37:15.355 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T06:37:15.404 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T06:37:15.405 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T06:37:15.410 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T06:37:15.410 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T06:37:15.424 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T06:37:15.568 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T06:37:15.598 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T06:37:16.008 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T06:37:16.013 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T06:37:16.028 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T06:37:16.028 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T06:37:16.031 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T06:37:16.150 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T06:37:16.260 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T06:37:16.280 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T06:37:16.282 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T06:37:16.282 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T06:37:16.286 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T06:37:16.286 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T06:37:16.287 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-25T06:37:16.293 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-25T06:37:16.298 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T06:37:16.463 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T06:37:16.546 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-25T06:37:16.653 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-25T06:37:16.710 INFO:teuthology.orchestra.run.smithi088.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T06:37:16.742 INFO:teuthology.orchestra.run.smithi088.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T06:37:21.133 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 13.3 MB in 3s (5,141 kB/s) 2024-04-25T06:37:22.471 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-04-25T06:37:22.489 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T06:37:22.592 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-04-25T06:37:22.792 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2024-04-25T06:37:22.793 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2024-04-25T06:37:22.953 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:37:22.954 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:37:22.954 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:37:22.954 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:37:22.954 INFO:teuthology.orchestra.run.smithi088.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T06:37:22.954 INFO:teuthology.orchestra.run.smithi088.stdout: python2.7-minimal 2024-04-25T06:37:22.954 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:37:22.955 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2024-04-25T06:37:22.955 INFO:teuthology.orchestra.run.smithi088.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T06:37:22.956 INFO:teuthology.orchestra.run.smithi088.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T06:37:22.956 INFO:teuthology.orchestra.run.smithi088.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T06:37:22.956 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2024-04-25T06:37:22.956 INFO:teuthology.orchestra.run.smithi088.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T06:37:22.957 INFO:teuthology.orchestra.run.smithi088.stdout:Recommended packages: 2024-04-25T06:37:22.957 INFO:teuthology.orchestra.run.smithi088.stdout: thermald 2024-04-25T06:37:23.008 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2024-04-25T06:37:23.008 INFO:teuthology.orchestra.run.smithi088.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T06:37:23.009 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T06:37:23.009 INFO:teuthology.orchestra.run.smithi088.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T06:37:23.010 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages will be upgraded: 2024-04-25T06:37:23.011 INFO:teuthology.orchestra.run.smithi088.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T06:37:23.191 INFO:teuthology.orchestra.run.smithi088.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T06:37:23.191 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 113 MB of archives. 2024-04-25T06:37:23.191 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T06:37:23.191 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T06:37:24.614 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T06:37:24.944 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T06:37:27.130 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T06:37:27.130 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T06:37:27.131 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T06:37:27.904 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T06:37:27.973 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T06:37:28.397 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 113 MB in 5s (22.9 MB/s) 2024-04-25T06:37:28.574 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T06:37:30.545 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T06:37:30.549 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:37:30.623 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:33.358 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T06:37:33.379 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:37:33.457 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:33.894 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T06:37:33.914 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:37:33.952 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:40.259 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:37:40.378 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:37:40.651 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:37:40.781 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:37:41.067 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T06:37:41.089 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T06:37:41.133 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T06:37:46.431 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T06:37:46.468 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:37:46.506 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:49.602 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:37:50.467 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:37:50.791 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T06:37:50.913 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:51.031 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T06:37:51.159 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:52.481 INFO:teuthology.orchestra.run.smithi088.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:37:52.481 INFO:teuthology.orchestra.run.smithi088.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T06:37:52.481 INFO:teuthology.orchestra.run.smithi088.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:37:52.481 INFO:teuthology.orchestra.run.smithi088.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T06:37:52.644 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:54.055 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:55.430 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T06:37:56.348 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T06:37:56.449 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:37:56.502 INFO:teuthology.orchestra.run.smithi088.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T06:37:56.502 INFO:teuthology.orchestra.run.smithi088.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:38:10.369 INFO:teuthology.orchestra.run.smithi088.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T06:38:10.369 INFO:teuthology.orchestra.run.smithi088.stdout:Sourcing file `/etc/default/grub' 2024-04-25T06:38:10.385 INFO:teuthology.orchestra.run.smithi088.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T06:38:10.415 INFO:teuthology.orchestra.run.smithi088.stdout:Generating grub configuration file ... 2024-04-25T06:38:10.730 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:38:10.749 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:38:11.139 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:38:11.145 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T06:38:11.212 INFO:teuthology.orchestra.run.smithi088.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T06:38:11.217 INFO:teuthology.orchestra.run.smithi088.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T06:38:11.399 INFO:teuthology.orchestra.run.smithi088.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T06:38:11.399 INFO:teuthology.orchestra.run.smithi088.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T06:38:11.399 INFO:teuthology.orchestra.run.smithi088.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T06:38:11.437 INFO:teuthology.orchestra.run.smithi088.stdout:done 2024-04-25T06:38:11.911 DEBUG:teuthology.orchestra.run.smithi088:> dpkg -s linux-image-generic 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Package: linux-image-generic 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Status: install ok installed 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Priority: optional 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Section: kernel 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Installed-Size: 21 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: amd64 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Source: linux-meta 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Version: 5.15.0.105.102 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Recommends: thermald 2024-04-25T06:38:11.944 INFO:teuthology.orchestra.run.smithi088.stdout:Description: Generic Linux kernel image 2024-04-25T06:38:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: This package will always depend on the latest generic kernel image 2024-04-25T06:38:11.945 INFO:teuthology.orchestra.run.smithi088.stdout: available. 2024-04-25T06:38:11.945 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T06:38:11.945 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T06:38:11.945 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T06:38:11.945 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-25T06:38:11.945 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T06:38:11.945 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get clean 2024-04-25T06:38:12.125 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2024-04-25T06:38:12.254 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:38:12.265 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T06:38:12.285 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T06:38:12.370 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T06:38:13.582 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-04-25T06:38:13.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T06:38:13.659 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-04-25T06:38:13.871 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2024-04-25T06:38:13.872 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2024-04-25T06:38:14.042 INFO:teuthology.orchestra.run.smithi088.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T06:38:14.042 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:38:14.042 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:38:14.042 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:38:14.042 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:38:14.042 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T06:38:14.043 INFO:teuthology.orchestra.run.smithi088.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T06:38:14.043 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:38:14.080 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T06:38:14.081 DEBUG:teuthology.orchestra.run.smithi088:> dpkg -s linux-image-generic 2024-04-25T06:38:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Package: linux-image-generic 2024-04-25T06:38:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Status: install ok installed 2024-04-25T06:38:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Priority: optional 2024-04-25T06:38:14.097 INFO:teuthology.orchestra.run.smithi088.stdout:Section: kernel 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Installed-Size: 21 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: amd64 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Source: linux-meta 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Version: 5.15.0.105.102 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Recommends: thermald 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Description: Generic Linux kernel image 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout: This package will always depend on the latest generic kernel image 2024-04-25T06:38:14.098 INFO:teuthology.orchestra.run.smithi088.stdout: available. 2024-04-25T06:38:14.099 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-04-25T06:38:14.143 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.MDRXzAhRn2 2024-04-25T06:38:14.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub/grub.cfg /tmp/tmp.MDRXzAhRn2 2024-04-25T06:38:14.206 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.MDRXzAhRn2 2024-04-25T06:38:14.324 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.MDRXzAhRn2 is 10KB 2024-04-25T06:38:14.337 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.MDRXzAhRn2 2024-04-25T06:38:14.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T06:38:14.399 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-25T06:38:14.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T06:38:14.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T06:38:14.467 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T06:38:14.467 DEBUG:teuthology.orchestra.run.smithi088:> sudo update-grub 2024-04-25T06:38:15.767 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub' 2024-04-25T06:38:15.768 INFO:teuthology.orchestra.run.smithi088.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T06:38:15.771 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-04-25T06:38:15.969 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:38:15.973 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:38:16.206 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:38:16.211 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T06:38:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T06:38:16.256 INFO:teuthology.orchestra.run.smithi088.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T06:38:16.365 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T06:38:16.366 INFO:teuthology.orchestra.run.smithi088.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T06:38:16.366 INFO:teuthology.orchestra.run.smithi088.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T06:38:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-04-25T06:38:16.394 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-04-25T06:38:46.426 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-25T06:38:46.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-25T06:38:46.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:39:21.119 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-25T06:39:30.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-25T06:39:30.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:39:33.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-25T06:39:45.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-25T06:39:45.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:40:03.615 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-25T06:40:18.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-25T06:40:18.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:40:18.949 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-04-25T06:40:19.514 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-25T06:40:19.514 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-25T06:40:19.514 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-04-25T06:40:19.560 INFO:teuthology.orchestra.run.smithi088.stdout:5.15.0-105-generic 2024-04-25T06:40:19.560 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T06:40:19.560 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T06:40:19.560 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T06:40:20.562 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T06:40:20.562 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T06:40:20.627 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-04-25T06:40:20.658 DEBUG:teuthology.parallel:result is None 2024-04-25T06:40:20.658 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T06:40:20.665 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T06:40:20.665 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T06:40:20.671 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T06:40:20.680 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T06:40:20.696 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T06:40:20.696 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T06:40:20.728 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T06:40:20.734 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T06:40:20.734 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T06:40:20.783 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:40:20.792 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:40:20.793 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T06:40:20.800 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T06:40:20.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T06:40:20.848 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T06:40:20.856 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T06:40:20.857 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T06:40:20.896 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T06:40:20.944 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T06:40:20.992 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-25T06:40:20.992 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T06:40:21.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-04-25T06:40:21.142 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T06:40:21.151 INFO:teuthology.task.internal:Starting timer... 2024-04-25T06:40:21.151 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T06:40:21.164 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T06:40:21.172 DEBUG:teuthology.task.selinux:Excluding smithi088: OS 'ubuntu' does not support SELinux 2024-04-25T06:40:21.172 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T06:40:21.172 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-25T06:40:21.172 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T06:40:21.172 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T06:40:21.181 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T06:40:21.197 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T06:40:21.198 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T06:47:12.366 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com')] 2024-04-25T06:47:12.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-25T06:47:12.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:47:12.453 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-04-25T06:47:12.530 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-25T06:47:12.530 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T06:47:12.543 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T06:47:12.543 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T06:47:12.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T06:47:12.597 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T06:47:12.597 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Command line: ntpd -gq 2024-04-25T06:47:12.597 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: ---------------------------------------------------- 2024-04-25T06:47:12.597 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T06:47:12.598 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T06:47:12.598 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: corporation. Support and training for ntp-4 are 2024-04-25T06:47:12.598 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: available at https://www.nwtime.org/support 2024-04-25T06:47:12.598 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: ---------------------------------------------------- 2024-04-25T06:47:12.598 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: proto: precision = 0.060 usec (-24) 2024-04-25T06:47:12.599 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: basedate set to 2022-02-04 2024-04-25T06:47:12.599 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: gps base set to 2022-02-06 (week 2196) 2024-04-25T06:47:12.599 INFO:teuthology.orchestra.run.smithi088.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T06:47:12.599 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T06:47:12.600 INFO:teuthology.orchestra.run.smithi088.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T06:47:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T06:47:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T06:47:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T06:47:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T06:47:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Listen normally on 3 enp3s0f1 172.21.15.88:123 2024-04-25T06:47:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Listen normally on 4 lo [::1]:123 2024-04-25T06:47:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d42d%5]:123 2024-04-25T06:47:12.602 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:12 ntpd[17228]: Listening on routing socket on fd #22 for interface updates 2024-04-25T06:47:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:14 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:14 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:14 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:14 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:14 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:14 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:14 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:14 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:15 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:15 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:15 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:15 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:15 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:15.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:15 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:16 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:16 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:16 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:16 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:16 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:16.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:16 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:17.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:17 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:18.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:18 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:18.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:18 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:18.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:18 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:18.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:18 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:18.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:18 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:18.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:18 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:19.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:19.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:19.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:19 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:20.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:20 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:20.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:20 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:20.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:20 ntpd[17228]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T06:47:20.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:20 ntpd[17228]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T06:47:20.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:20 ntpd[17228]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T06:47:20.598 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:20 ntpd[17228]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T06:47:20.599 INFO:teuthology.orchestra.run.smithi088.stdout:25 Apr 06:47:20 ntpd[17228]: ntpd: time slew -0.000670 s 2024-04-25T06:47:20.599 INFO:teuthology.orchestra.run.smithi088.stdout:ntpd: time slew -0.000670s 2024-04-25T06:47:20.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:20 ntpd[17228]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T06:47:20.599 INFO:teuthology.orchestra.run.smithi088.stderr:25 Apr 06:47:20 ntpd[17228]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T06:47:20.703 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T06:47:20.703 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2024-04-25T06:47:20.703 INFO:teuthology.orchestra.run.smithi088.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T06:47:20.703 INFO:teuthology.orchestra.run.smithi088.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T06:47:20.703 INFO:teuthology.orchestra.run.smithi088.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T06:47:20.703 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T06:47:20.704 INFO:teuthology.run_tasks:Running task install... 2024-04-25T06:47:20.713 DEBUG:teuthology.task.install:project ceph 2024-04-25T06:47:20.713 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'}} 2024-04-25T06:47:20.713 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'} 2024-04-25T06:47:20.713 INFO:teuthology.task.install:Using flavor: default 2024-04-25T06:47:20.719 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T06:47:20.719 INFO:teuthology.task.install:extra packages: [] 2024-04-25T06:47:20.719 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-key list | grep Ceph 2024-04-25T06:47:20.809 INFO:teuthology.orchestra.run.smithi088.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T06:47:20.836 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T06:47:20.836 INFO:teuthology.orchestra.run.smithi088.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T06:47:20.836 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-25T06:47:20.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T06:47:20.997 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default/ 2024-04-25T06:47:20.997 INFO:teuthology.task.install.deb:Package version is 19.0.0-3227-gc37f63b3-1jammy 2024-04-25T06:47:21.127 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-25T06:47:21.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T06:47:21.141 DEBUG:teuthology.orchestra.run.smithi088:> sudo apt-get update 2024-04-25T06:47:21.313 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:47:21.327 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T06:47:21.348 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T06:47:21.362 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T06:47:21.383 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T06:47:21.405 INFO:teuthology.orchestra.run.smithi088.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T06:47:21.426 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-04-25T06:47:21.431 INFO:teuthology.orchestra.run.smithi088.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T06:47:21.452 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-04-25T06:47:21.658 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 29.3 kB in 0s (85.6 kB/s) 2024-04-25T06:47:22.596 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-04-25T06:47:22.612 DEBUG:teuthology.orchestra.run.smithi088:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3227-gc37f63b3-1jammy cephadm=19.0.0-3227-gc37f63b3-1jammy ceph-mds=19.0.0-3227-gc37f63b3-1jammy ceph-mgr=19.0.0-3227-gc37f63b3-1jammy ceph-common=19.0.0-3227-gc37f63b3-1jammy ceph-fuse=19.0.0-3227-gc37f63b3-1jammy ceph-test=19.0.0-3227-gc37f63b3-1jammy ceph-volume=19.0.0-3227-gc37f63b3-1jammy radosgw=19.0.0-3227-gc37f63b3-1jammy python3-rados=19.0.0-3227-gc37f63b3-1jammy python3-rgw=19.0.0-3227-gc37f63b3-1jammy python3-cephfs=19.0.0-3227-gc37f63b3-1jammy python3-rbd=19.0.0-3227-gc37f63b3-1jammy libcephfs2=19.0.0-3227-gc37f63b3-1jammy libcephfs-dev=19.0.0-3227-gc37f63b3-1jammy librados2=19.0.0-3227-gc37f63b3-1jammy librbd1=19.0.0-3227-gc37f63b3-1jammy rbd-fuse=19.0.0-3227-gc37f63b3-1jammy 2024-04-25T06:47:22.674 INFO:teuthology.orchestra.run.smithi088.stdout:Reading package lists... 2024-04-25T06:47:22.882 INFO:teuthology.orchestra.run.smithi088.stdout:Building dependency tree... 2024-04-25T06:47:22.883 INFO:teuthology.orchestra.run.smithi088.stdout:Reading state information... 2024-04-25T06:47:23.080 INFO:teuthology.orchestra.run.smithi088.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:47:23.080 INFO:teuthology.orchestra.run.smithi088.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:47:23.080 INFO:teuthology.orchestra.run.smithi088.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:47:23.080 INFO:teuthology.orchestra.run.smithi088.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:47:23.081 INFO:teuthology.orchestra.run.smithi088.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T06:47:23.081 INFO:teuthology.orchestra.run.smithi088.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T06:47:23.082 INFO:teuthology.orchestra.run.smithi088.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:47:23.082 INFO:teuthology.orchestra.run.smithi088.stdout:The following additional packages will be installed: 2024-04-25T06:47:23.082 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T06:47:23.082 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T06:47:23.082 INFO:teuthology.orchestra.run.smithi088.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T06:47:23.082 INFO:teuthology.orchestra.run.smithi088.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T06:47:23.083 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T06:47:23.083 INFO:teuthology.orchestra.run.smithi088.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T06:47:23.083 INFO:teuthology.orchestra.run.smithi088.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T06:47:23.083 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T06:47:23.083 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T06:47:23.083 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T06:47:23.083 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-25T06:47:23.084 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet zip 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout:Suggested packages: 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout:Recommended packages: 2024-04-25T06:47:23.085 INFO:teuthology.orchestra.run.smithi088.stdout: btrfs-tools 2024-04-25T06:47:23.184 INFO:teuthology.orchestra.run.smithi088.stdout:The following NEW packages will be installed: 2024-04-25T06:47:23.184 INFO:teuthology.orchestra.run.smithi088.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T06:47:23.185 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T06:47:23.185 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-25T06:47:23.185 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-25T06:47:23.185 INFO:teuthology.orchestra.run.smithi088.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-25T06:47:23.185 INFO:teuthology.orchestra.run.smithi088.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-25T06:47:23.186 INFO:teuthology.orchestra.run.smithi088.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-25T06:47:23.186 INFO:teuthology.orchestra.run.smithi088.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-25T06:47:23.186 INFO:teuthology.orchestra.run.smithi088.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-25T06:47:23.186 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-25T06:47:23.186 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-25T06:47:23.186 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T06:47:23.186 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T06:47:23.186 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-25T06:47:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-25T06:47:23.267 INFO:teuthology.orchestra.run.smithi088.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T06:47:23.267 INFO:teuthology.orchestra.run.smithi088.stdout:Need to get 181 MB of archives. 2024-04-25T06:47:23.267 INFO:teuthology.orchestra.run.smithi088.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-25T06:47:23.267 INFO:teuthology.orchestra.run.smithi088.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T06:47:23.338 INFO:teuthology.orchestra.run.smithi088.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3227-gc37f63b3-1jammy [3,556 kB] 2024-04-25T06:47:23.441 INFO:teuthology.orchestra.run.smithi088.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T06:47:23.445 INFO:teuthology.orchestra.run.smithi088.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T06:47:23.475 INFO:teuthology.orchestra.run.smithi088.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T06:47:23.546 INFO:teuthology.orchestra.run.smithi088.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T06:47:23.550 INFO:teuthology.orchestra.run.smithi088.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T06:47:23.561 INFO:teuthology.orchestra.run.smithi088.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3227-gc37f63b3-1jammy [3,224 kB] 2024-04-25T06:47:23.562 INFO:teuthology.orchestra.run.smithi088.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T06:47:23.567 INFO:teuthology.orchestra.run.smithi088.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T06:47:23.567 INFO:teuthology.orchestra.run.smithi088.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T06:47:23.568 INFO:teuthology.orchestra.run.smithi088.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T06:47:23.577 INFO:teuthology.orchestra.run.smithi088.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T06:47:23.579 INFO:teuthology.orchestra.run.smithi088.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T06:47:23.581 INFO:teuthology.orchestra.run.smithi088.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T06:47:23.583 INFO:teuthology.orchestra.run.smithi088.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T06:47:23.591 INFO:teuthology.orchestra.run.smithi088.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T06:47:23.593 INFO:teuthology.orchestra.run.smithi088.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T06:47:23.595 INFO:teuthology.orchestra.run.smithi088.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T06:47:23.597 INFO:teuthology.orchestra.run.smithi088.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T06:47:23.612 INFO:teuthology.orchestra.run.smithi088.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T06:47:23.612 INFO:teuthology.orchestra.run.smithi088.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T06:47:23.613 INFO:teuthology.orchestra.run.smithi088.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T06:47:23.614 INFO:teuthology.orchestra.run.smithi088.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T06:47:23.615 INFO:teuthology.orchestra.run.smithi088.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T06:47:23.615 INFO:teuthology.orchestra.run.smithi088.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T06:47:23.633 INFO:teuthology.orchestra.run.smithi088.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T06:47:23.634 INFO:teuthology.orchestra.run.smithi088.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T06:47:23.634 INFO:teuthology.orchestra.run.smithi088.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T06:47:23.637 INFO:teuthology.orchestra.run.smithi088.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T06:47:23.654 INFO:teuthology.orchestra.run.smithi088.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T06:47:23.654 INFO:teuthology.orchestra.run.smithi088.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T06:47:23.655 INFO:teuthology.orchestra.run.smithi088.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T06:47:23.656 INFO:teuthology.orchestra.run.smithi088.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T06:47:23.656 INFO:teuthology.orchestra.run.smithi088.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T06:47:23.656 INFO:teuthology.orchestra.run.smithi088.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T06:47:23.659 INFO:teuthology.orchestra.run.smithi088.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3227-gc37f63b3-1jammy [712 kB] 2024-04-25T06:47:23.675 INFO:teuthology.orchestra.run.smithi088.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T06:47:23.676 INFO:teuthology.orchestra.run.smithi088.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T06:47:23.676 INFO:teuthology.orchestra.run.smithi088.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T06:47:23.681 INFO:teuthology.orchestra.run.smithi088.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T06:47:23.682 INFO:teuthology.orchestra.run.smithi088.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3227-gc37f63b3-1jammy [356 kB] 2024-04-25T06:47:23.693 INFO:teuthology.orchestra.run.smithi088.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3227-gc37f63b3-1jammy [32.8 kB] 2024-04-25T06:47:23.694 INFO:teuthology.orchestra.run.smithi088.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3227-gc37f63b3-1jammy [180 kB] 2024-04-25T06:47:23.695 INFO:teuthology.orchestra.run.smithi088.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T06:47:23.696 INFO:teuthology.orchestra.run.smithi088.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T06:47:23.697 INFO:teuthology.orchestra.run.smithi088.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T06:47:23.698 INFO:teuthology.orchestra.run.smithi088.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T06:47:23.700 INFO:teuthology.orchestra.run.smithi088.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T06:47:23.701 INFO:teuthology.orchestra.run.smithi088.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T06:47:23.701 INFO:teuthology.orchestra.run.smithi088.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3227-gc37f63b3-1jammy [67.1 kB] 2024-04-25T06:47:23.703 INFO:teuthology.orchestra.run.smithi088.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3227-gc37f63b3-1jammy [330 kB] 2024-04-25T06:47:23.713 INFO:teuthology.orchestra.run.smithi088.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3227-gc37f63b3-1jammy [6,893 kB] 2024-04-25T06:47:23.716 INFO:teuthology.orchestra.run.smithi088.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T06:47:23.717 INFO:teuthology.orchestra.run.smithi088.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T06:47:23.719 INFO:teuthology.orchestra.run.smithi088.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T06:47:23.720 INFO:teuthology.orchestra.run.smithi088.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T06:47:23.737 INFO:teuthology.orchestra.run.smithi088.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T06:47:23.740 INFO:teuthology.orchestra.run.smithi088.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T06:47:23.820 INFO:teuthology.orchestra.run.smithi088.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T06:47:23.821 INFO:teuthology.orchestra.run.smithi088.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T06:47:23.822 INFO:teuthology.orchestra.run.smithi088.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T06:47:23.823 INFO:teuthology.orchestra.run.smithi088.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T06:47:23.824 INFO:teuthology.orchestra.run.smithi088.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T06:47:23.927 INFO:teuthology.orchestra.run.smithi088.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T06:47:23.931 INFO:teuthology.orchestra.run.smithi088.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T06:47:23.933 INFO:teuthology.orchestra.run.smithi088.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T06:47:23.958 INFO:teuthology.orchestra.run.smithi088.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T06:47:23.958 INFO:teuthology.orchestra.run.smithi088.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T06:47:23.959 INFO:teuthology.orchestra.run.smithi088.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T06:47:23.959 INFO:teuthology.orchestra.run.smithi088.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T06:47:23.960 INFO:teuthology.orchestra.run.smithi088.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T06:47:23.960 INFO:teuthology.orchestra.run.smithi088.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T06:47:23.961 INFO:teuthology.orchestra.run.smithi088.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T06:47:23.972 INFO:teuthology.orchestra.run.smithi088.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3227-gc37f63b3-1jammy [112 kB] 2024-04-25T06:47:23.972 INFO:teuthology.orchestra.run.smithi088.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T06:47:23.976 INFO:teuthology.orchestra.run.smithi088.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T06:47:23.978 INFO:teuthology.orchestra.run.smithi088.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3227-gc37f63b3-1jammy [466 kB] 2024-04-25T06:47:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3227-gc37f63b3-1jammy [25.8 MB] 2024-04-25T06:47:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T06:47:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T06:47:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T06:47:23.999 INFO:teuthology.orchestra.run.smithi088.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T06:47:24.002 INFO:teuthology.orchestra.run.smithi088.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T06:47:24.003 INFO:teuthology.orchestra.run.smithi088.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T06:47:24.004 INFO:teuthology.orchestra.run.smithi088.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T06:47:24.010 INFO:teuthology.orchestra.run.smithi088.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T06:47:24.013 INFO:teuthology.orchestra.run.smithi088.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T06:47:24.017 INFO:teuthology.orchestra.run.smithi088.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T06:47:24.034 INFO:teuthology.orchestra.run.smithi088.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T06:47:24.035 INFO:teuthology.orchestra.run.smithi088.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T06:47:24.035 INFO:teuthology.orchestra.run.smithi088.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T06:47:24.037 INFO:teuthology.orchestra.run.smithi088.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T06:47:24.038 INFO:teuthology.orchestra.run.smithi088.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T06:47:24.046 INFO:teuthology.orchestra.run.smithi088.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T06:47:24.046 INFO:teuthology.orchestra.run.smithi088.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T06:47:24.047 INFO:teuthology.orchestra.run.smithi088.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T06:47:24.054 INFO:teuthology.orchestra.run.smithi088.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T06:47:24.055 INFO:teuthology.orchestra.run.smithi088.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T06:47:24.932 INFO:teuthology.orchestra.run.smithi088.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3227-gc37f63b3-1jammy [5,092 kB] 2024-04-25T06:47:25.009 INFO:teuthology.orchestra.run.smithi088.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3227-gc37f63b3-1jammy [244 kB] 2024-04-25T06:47:25.013 INFO:teuthology.orchestra.run.smithi088.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3227-gc37f63b3-1jammy [124 kB] 2024-04-25T06:47:25.017 INFO:teuthology.orchestra.run.smithi088.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3227-gc37f63b3-1jammy [1,526 kB] 2024-04-25T06:47:25.067 INFO:teuthology.orchestra.run.smithi088.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3227-gc37f63b3-1jammy [6,257 kB] 2024-04-25T06:47:25.278 INFO:teuthology.orchestra.run.smithi088.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3227-gc37f63b3-1jammy [22.8 MB] 2024-04-25T06:47:26.118 INFO:teuthology.orchestra.run.smithi088.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3227-gc37f63b3-1jammy [14.1 kB] 2024-04-25T06:47:26.118 INFO:teuthology.orchestra.run.smithi088.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3227-gc37f63b3-1jammy [870 kB] 2024-04-25T06:47:26.135 INFO:teuthology.orchestra.run.smithi088.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3227-gc37f63b3-1jammy [2,437 kB] 2024-04-25T06:47:26.177 INFO:teuthology.orchestra.run.smithi088.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3227-gc37f63b3-1jammy [791 kB] 2024-04-25T06:47:26.186 INFO:teuthology.orchestra.run.smithi088.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3227-gc37f63b3-1jammy [148 kB] 2024-04-25T06:47:26.190 INFO:teuthology.orchestra.run.smithi088.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3227-gc37f63b3-1jammy [2,297 kB] 2024-04-25T06:47:26.235 INFO:teuthology.orchestra.run.smithi088.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3227-gc37f63b3-1jammy [8,625 kB] 2024-04-25T06:47:26.519 INFO:teuthology.orchestra.run.smithi088.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3227-gc37f63b3-1jammy [14.2 kB] 2024-04-25T06:47:26.519 INFO:teuthology.orchestra.run.smithi088.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3227-gc37f63b3-1jammy [51.6 MB] 2024-04-25T06:47:28.567 INFO:teuthology.orchestra.run.smithi088.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3227-gc37f63b3-1jammy [124 kB] 2024-04-25T06:47:28.569 INFO:teuthology.orchestra.run.smithi088.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3227-gc37f63b3-1jammy [39.0 kB] 2024-04-25T06:47:28.570 INFO:teuthology.orchestra.run.smithi088.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3227-gc37f63b3-1jammy [13.6 MB] 2024-04-25T06:47:28.908 INFO:teuthology.orchestra.run.smithi088.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3227-gc37f63b3-1jammy [91.4 kB] 2024-04-25T06:47:29.534 INFO:teuthology.orchestra.run.smithi088.stdout:Fetched 181 MB in 6s (31.3 MB/s) 2024-04-25T06:47:29.715 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T06:47:29.816 INFO:teuthology.orchestra.run.smithi088.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T06:47:29.821 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T06:47:29.905 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T06:47:30.243 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T06:47:30.269 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T06:47:30.309 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T06:47:30.612 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T06:47:30.638 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T06:47:30.670 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T06:47:31.082 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T06:47:31.109 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T06:47:31.157 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T06:47:31.560 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T06:47:31.586 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T06:47:31.626 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T06:47:31.913 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T06:47:31.939 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T06:47:31.971 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T06:47:32.318 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T06:47:32.344 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T06:47:32.385 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T06:47:32.679 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librados2. 2024-04-25T06:47:32.694 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../007-librados2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:32.729 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librados2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:33.165 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librbd1. 2024-04-25T06:47:33.191 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../008-librbd1_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:33.248 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librbd1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:33.701 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T06:47:33.728 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:33.768 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:34.003 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rados. 2024-04-25T06:47:34.018 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:34.053 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rados (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:34.389 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T06:47:34.416 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T06:47:34.456 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-argparse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:34.708 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T06:47:34.735 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:34.775 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cephfs (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:35.086 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T06:47:35.112 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T06:47:35.152 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:35.464 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T06:47:35.490 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T06:47:35.530 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T06:47:36.000 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T06:47:36.027 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T06:47:36.066 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T06:47:36.327 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T06:47:36.343 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:36.386 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rbd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:36.713 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T06:47:36.740 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T06:47:36.780 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T06:47:37.057 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T06:47:37.084 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T06:47:37.124 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T06:47:37.394 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T06:47:37.420 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T06:47:37.461 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T06:47:37.780 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package lua5.1. 2024-04-25T06:47:37.806 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T06:47:37.846 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T06:47:38.166 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package lua-any. 2024-04-25T06:47:38.192 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-25T06:47:38.232 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T06:47:38.443 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package zip. 2024-04-25T06:47:38.459 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-25T06:47:38.493 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T06:47:38.788 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package unzip. 2024-04-25T06:47:38.815 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T06:47:38.855 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T06:47:39.258 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package luarocks. 2024-04-25T06:47:39.285 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T06:47:39.325 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T06:47:39.707 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package librgw2. 2024-04-25T06:47:39.734 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../025-librgw2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:39.774 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking librgw2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:40.285 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T06:47:40.300 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:40.335 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:40.679 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T06:47:40.705 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T06:47:40.746 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T06:47:41.057 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T06:47:41.084 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:41.123 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libradosstriper1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:41.535 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-common. 2024-04-25T06:47:41.562 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:41.593 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:42.949 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-base. 2024-04-25T06:47:42.975 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:43.024 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-base (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:43.586 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T06:47:43.613 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T06:47:43.652 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T06:47:43.938 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T06:47:43.965 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T06:47:44.022 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T06:47:44.341 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T06:47:44.368 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T06:47:44.416 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T06:47:44.677 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T06:47:44.704 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T06:47:44.744 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T06:47:45.047 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T06:47:45.073 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T06:47:45.113 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T06:47:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T06:47:45.442 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T06:47:45.482 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T06:47:45.777 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-portend. 2024-04-25T06:47:45.804 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-25T06:47:45.835 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T06:47:46.113 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T06:47:46.140 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T06:47:46.180 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T06:47:46.632 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T06:47:46.659 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T06:47:46.699 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T06:47:47.011 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T06:47:47.038 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T06:47:47.077 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T06:47:47.389 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T06:47:47.415 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T06:47:47.455 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T06:47:47.766 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T06:47:47.793 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T06:47:47.833 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T06:47:48.122 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-mako. 2024-04-25T06:47:48.146 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T06:47:48.186 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T06:47:48.497 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T06:47:48.523 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T06:47:48.563 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T06:47:48.875 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T06:47:48.901 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T06:47:48.941 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T06:47:49.219 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webob. 2024-04-25T06:47:49.245 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T06:47:49.286 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T06:47:49.580 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T06:47:49.606 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T06:47:49.641 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T06:47:49.900 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T06:47:49.926 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T06:47:49.966 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T06:47:50.269 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-paste. 2024-04-25T06:47:50.295 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T06:47:50.336 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T06:47:50.655 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T06:47:50.670 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T06:47:50.705 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T06:47:51.008 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T06:47:51.034 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T06:47:51.075 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T06:47:51.411 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T06:47:51.438 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T06:47:51.477 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T06:47:51.947 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T06:47:51.974 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T06:47:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T06:47:52.377 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T06:47:52.404 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T06:47:52.443 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T06:47:52.704 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-idna. 2024-04-25T06:47:52.720 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-25T06:47:52.754 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T06:47:53.065 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T06:47:53.092 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T06:47:53.132 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T06:47:53.452 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests. 2024-04-25T06:47:53.479 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T06:47:53.518 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T06:47:53.854 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T06:47:53.882 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T06:47:53.921 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T06:47:54.216 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T06:47:54.243 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T06:47:54.282 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:54.629 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T06:47:54.655 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:54.695 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:54.998 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T06:47:55.025 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T06:47:55.064 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T06:47:55.350 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T06:47:55.378 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T06:47:55.417 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T06:47:55.762 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T06:47:55.789 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:55.828 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:56.164 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T06:47:56.179 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:56.214 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mon (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:57.577 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T06:47:57.603 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:57.643 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-osd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:58.788 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph. 2024-04-25T06:47:58.803 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../066-ceph_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:58.846 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:59.124 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T06:47:59.150 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:59.182 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:47:59.585 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T06:47:59.611 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:47:59.651 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mds (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:00.046 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package cephadm. 2024-04-25T06:48:00.073 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../069-cephadm_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:48:00.112 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:00.432 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T06:48:00.458 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T06:48:00.498 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T06:48:00.809 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T06:48:00.836 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T06:48:00.876 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T06:48:01.922 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-babel. 2024-04-25T06:48:01.949 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T06:48:01.989 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T06:48:02.309 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T06:48:02.335 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T06:48:02.375 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T06:48:02.728 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T06:48:02.755 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T06:48:02.794 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:03.148 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T06:48:03.174 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T06:48:03.214 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T06:48:03.509 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-routes. 2024-04-25T06:48:03.535 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T06:48:03.575 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T06:48:03.920 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T06:48:03.946 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T06:48:03.986 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:05.216 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T06:48:05.243 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T06:48:05.283 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T06:48:05.811 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T06:48:05.838 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T06:48:05.877 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T06:48:06.222 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T06:48:06.250 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T06:48:06.280 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T06:48:06.542 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T06:48:06.569 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T06:48:06.608 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T06:48:07.491 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T06:48:07.519 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T06:48:07.643 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:08.522 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T06:48:08.549 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T06:48:08.588 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T06:48:08.891 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T06:48:08.918 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T06:48:08.974 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T06:48:09.321 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T06:48:09.346 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T06:48:09.385 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T06:48:09.680 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T06:48:09.707 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-25T06:48:09.738 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T06:48:10.027 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T06:48:10.054 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T06:48:10.093 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T06:48:10.413 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T06:48:10.440 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T06:48:10.480 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T06:48:10.774 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T06:48:10.790 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T06:48:10.824 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T06:48:11.103 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T06:48:11.130 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T06:48:11.161 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T06:48:11.774 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T06:48:11.802 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T06:48:11.840 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:12.127 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T06:48:12.143 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T06:48:12.177 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T06:48:12.513 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T06:48:12.540 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T06:48:12.579 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T06:48:12.857 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package jq. 2024-04-25T06:48:12.884 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T06:48:12.923 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T06:48:13.168 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package socat. 2024-04-25T06:48:13.195 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T06:48:13.235 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T06:48:13.513 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T06:48:13.528 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T06:48:13.562 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T06:48:13.892 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-test. 2024-04-25T06:48:13.919 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:48:13.958 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-test (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:16.146 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T06:48:16.173 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T06:48:16.212 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking ceph-volume (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:16.541 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T06:48:16.568 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:48:16.607 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking libcephfs-dev (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:16.927 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T06:48:16.954 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T06:48:16.993 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T06:48:17.474 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T06:48:17.501 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T06:48:17.547 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T06:48:18.110 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T06:48:18.137 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T06:48:18.176 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T06:48:18.598 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package pkg-config. 2024-04-25T06:48:18.626 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T06:48:18.664 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T06:48:18.892 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T06:48:18.908 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T06:48:18.941 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T06:48:19.337 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T06:48:19.364 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T06:48:19.395 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T06:48:19.672 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T06:48:19.700 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T06:48:19.739 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T06:48:20.084 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T06:48:20.111 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T06:48:20.150 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T06:48:20.462 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T06:48:20.489 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T06:48:20.528 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T06:48:20.856 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-py. 2024-04-25T06:48:20.883 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-25T06:48:20.922 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T06:48:21.226 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T06:48:21.253 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T06:48:21.292 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T06:48:21.771 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T06:48:21.799 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T06:48:21.837 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T06:48:22.140 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-toml. 2024-04-25T06:48:22.168 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-25T06:48:22.206 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T06:48:22.484 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T06:48:22.512 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T06:48:22.551 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T06:48:22.904 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T06:48:22.932 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T06:48:22.970 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T06:48:23.423 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T06:48:23.451 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T06:48:23.490 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T06:48:24.194 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package radosgw. 2024-04-25T06:48:24.222 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../116-radosgw_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:48:24.260 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking radosgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:25.080 INFO:teuthology.orchestra.run.smithi088.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T06:48:25.108 INFO:teuthology.orchestra.run.smithi088.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T06:48:25.138 INFO:teuthology.orchestra.run.smithi088.stdout:Unpacking rbd-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:25.417 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T06:48:25.685 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T06:48:25.811 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T06:48:26.070 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T06:48:26.406 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T06:48:26.842 INFO:teuthology.orchestra.run.smithi088.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T06:48:26.920 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:27.505 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user cephadm....done 2024-04-25T06:48:27.695 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T06:48:28.155 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T06:48:28.414 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T06:48:28.540 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T06:48:28.791 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T06:48:29.034 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T06:48:29.160 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T06:48:29.470 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T06:48:29.822 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T06:48:30.073 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T06:48:30.340 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T06:48:30.579 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T06:48:30.835 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T06:48:31.094 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T06:48:31.437 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-argparse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:31.721 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T06:48:31.847 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T06:48:31.948 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T06:48:32.049 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T06:48:32.317 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T06:48:32.694 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T06:48:32.724 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T06:48:32.778 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T06:48:32.879 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T06:48:33.239 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T06:48:33.365 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T06:48:33.624 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T06:48:33.884 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T06:48:34.218 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T06:48:34.344 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T06:48:35.071 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T06:48:35.322 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T06:48:35.449 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T06:48:35.566 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T06:48:35.850 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T06:48:36.252 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T06:48:36.502 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T06:48:36.879 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T06:48:37.163 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T06:48:37.490 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T06:48:37.750 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T06:48:37.876 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:38.177 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T06:48:38.486 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T06:48:39.659 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T06:48:39.919 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T06:48:40.053 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T06:48:40.312 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T06:48:40.438 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T06:48:40.564 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T06:48:40.857 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T06:48:41.125 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T06:48:41.376 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T06:48:41.502 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T06:48:41.804 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T06:48:42.063 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T06:48:42.305 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T06:48:42.540 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T06:48:42.783 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T06:48:43.068 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T06:48:43.193 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T06:48:43.469 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T06:48:43.595 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T06:48:43.897 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T06:48:44.206 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T06:48:44.440 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T06:48:44.566 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T06:48:44.692 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T06:48:44.781 INFO:teuthology.orchestra.run.smithi088.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T06:48:45.019 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T06:48:45.162 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T06:48:45.563 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T06:48:45.814 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T06:48:45.940 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T06:48:46.191 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T06:48:46.292 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T06:48:46.593 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T06:48:46.870 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T06:48:47.146 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T06:48:47.271 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T06:48:47.539 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T06:48:47.815 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T06:48:48.152 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T06:48:48.537 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T06:48:48.813 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T06:48:49.149 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T06:48:49.259 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librados2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:49.360 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:49.469 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T06:48:49.780 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:49.906 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libradosstriper1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:50.032 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T06:48:50.317 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librbd1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:50.435 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-modules-core (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:50.544 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:50.738 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T06:48:50.738 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T06:48:51.195 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up libcephfs-dev (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:51.321 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rados (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:51.447 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up librgw2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:51.573 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T06:48:52.726 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rbd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:52.827 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up rbd-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:52.954 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-rgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:53.079 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up python3-cephfs (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:53.205 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:53.748 INFO:teuthology.orchestra.run.smithi088.stdout:Adding group ceph....done 2024-04-25T06:48:54.083 INFO:teuthology.orchestra.run.smithi088.stdout:Adding system user ceph....done 2024-04-25T06:48:54.243 INFO:teuthology.orchestra.run.smithi088.stdout:Setting system user ceph properties....done 2024-04-25T06:48:54.260 INFO:teuthology.orchestra.run.smithi088.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T06:48:54.398 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T06:48:54.694 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T06:48:55.131 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-test (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:55.266 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up radosgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:55.745 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T06:48:55.745 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T06:48:56.226 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-base (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:56.550 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T06:48:57.019 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mds (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:57.254 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T06:48:57.254 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T06:48:57.729 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:57.945 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T06:48:57.945 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T06:48:58.438 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-osd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:58.731 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T06:48:58.731 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T06:48:59.242 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:59.377 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:59.503 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mon (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:48:59.701 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T06:48:59.701 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T06:49:00.237 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:49:00.380 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:49:00.506 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-mgr-dashboard (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:49:00.657 INFO:teuthology.orchestra.run.smithi088.stdout:Setting up ceph-volume (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T06:49:01.034 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T06:49:01.253 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T06:49:01.380 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T06:49:02.481 INFO:teuthology.orchestra.run.smithi088.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T06:49:03.166 INFO:teuthology.orchestra.run.smithi088.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T06:49:03.172 DEBUG:teuthology.parallel:result is None 2024-04-25T06:49:03.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T06:49:03.323 DEBUG:teuthology.orchestra.run.smithi088:> dpkg-query -W -f '${Version}' ceph 2024-04-25T06:49:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:19.0.0-3227-gc37f63b3-1jammy 2024-04-25T06:49:03.342 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3227-gc37f63b3-1jammy 2024-04-25T06:49:03.342 INFO:teuthology.task.install:The correct ceph version 19.0.0-3227-gc37f63b3-1jammy is installed. 2024-04-25T06:49:03.344 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T06:49:03.344 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-25T06:49:03.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T06:49:03.396 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T06:49:03.396 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-25T06:49:03.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T06:49:03.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T06:49:03.506 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T06:49:03.506 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-25T06:49:03.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T06:49:03.558 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T06:49:03.610 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T06:49:03.610 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-25T06:49:03.610 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T06:49:03.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T06:49:03.719 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T06:49:03.733 INFO:tasks.workunit:Pulling workunits from ref a08304379f921e5e229e92c75ede8800059313ff 2024-04-25T06:49:03.733 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T06:49:03.733 INFO:tasks.workunit:timeout=3h 2024-04-25T06:49:03.733 INFO:tasks.workunit:cleanup=True 2024-04-25T06:49:03.734 DEBUG:teuthology.orchestra.run.smithi088:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T06:49:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:49:03.768 INFO:teuthology.orchestra.run.smithi088.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-25T06:49:03.768 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T06:49:03.815 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-25T06:49:03.816 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-25T06:49:03.864 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/NitzanMordhai/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a08304379f921e5e229e92c75ede8800059313ff 2024-04-25T06:49:03.914 INFO:tasks.workunit.client.0.smithi088.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T06:49:42.525 INFO:tasks.workunit.client.0.smithi088.stderr:Note: switching to 'a08304379f921e5e229e92c75ede8800059313ff'. 2024-04-25T06:49:42.525 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr: git switch -c 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr:Or undo this operation with: 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr: git switch - 2024-04-25T06:49:42.526 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T06:49:42.527 INFO:tasks.workunit.client.0.smithi088.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T06:49:42.527 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T06:49:42.527 INFO:tasks.workunit.client.0.smithi088.stderr:HEAD is now at a08304379f9 Merge pull request #56420 from rzarzynski/wip-tools-rados-get-with-offset 2024-04-25T06:49:42.528 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T06:49:42.577 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-25T06:49:42.577 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T06:49:42.623 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-04-25T06:49:42.623 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-04-25T06:49:42.624 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-04-25T06:49:42.690 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T06:49:42.724 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/bad-inc-map 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:56063 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:56063 2024-04-25T06:49:42.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-04-25T06:49:42.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-04-25T06:49:42.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none ' 2024-04-25T06:49:42.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:56063 ' 2024-04-25T06:49:42.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-04-25T06:49:42.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-04-25T06:49:42.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T06:49:42.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-04-25T06:49:42.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-04-25T06:49:42.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-04-25T06:49:42.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-04-25T06:49:42.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-04-25T06:49:42.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T06:49:42.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T06:49:42.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T06:49:42.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:49:42.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:49:42.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:49:42.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:49:42.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:49:42.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:49:42.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T06:49:42.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T06:49:42.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T06:49:42.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T06:49:42.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T06:49:42.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T06:49:42.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:49:42.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T06:49:42.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T06:49:42.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:49:42.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:49:42.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T06:49:42.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T06:49:42.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T06:49:42.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:49:42.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:49:42.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20256 2024-04-25T06:49:42.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T06:49:42.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T06:49:42.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-04-25T06:49:42.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T06:49:42.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:49:42.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:49:42.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20256 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-04-25T06:49:42.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T06:49:42.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T06:49:42.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T06:49:42.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-04-25T06:49:42.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-04-25T06:49:43.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T06:49:43.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:49:43.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:49:43.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:49:43.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:49:43.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:49:43.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20256/$cluster-$name.asok' 2024-04-25T06:49:43.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T06:49:43.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T06:49:43.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T06:49:43.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T06:49:43.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T06:49:43.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T06:49:43.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T06:49:43.492 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T06:49:43.492 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T06:49:43.492 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T06:49:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:49:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:49:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:49:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20256/ceph-mon.a.asok 2024-04-25T06:49:43.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T06:49:43.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20256/ceph-mon.a.asok config get fsid 2024-04-25T06:49:43.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T06:49:43.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T06:49:43.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T06:49:43.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20256/ceph-mon.a.asok 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T06:49:43.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20256/ceph-mon.a.asok config get mon_host 2024-04-25T06:49:43.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-04-25T06:49:43.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-04-25T06:49:43.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T06:49:43.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T06:49:43.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T06:49:43.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-04-25T06:49:43.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T06:49:44.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T06:49:44.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:49:44.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:49:44.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:49:44.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:49:44.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:49:44.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20256/$cluster-$name.asok' 2024-04-25T06:49:44.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T06:49:44.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T06:49:44.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-04-25T06:49:44.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T06:49:44.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:49:44.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T06:49:44.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:49:44.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-04-25T06:49:44.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 ' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:49:44.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20256/$cluster-$name.asok' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:49:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-04-25T06:49:44.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:49:44.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d72d565-b607-48cc-a487-e0ecb225826c 2024-04-25T06:49:44.761 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 4d72d565-b607-48cc-a487-e0ecb225826c 2024-04-25T06:49:44.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4d72d565-b607-48cc-a487-e0ecb225826c' 2024-04-25T06:49:44.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:49:44.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAI/Slm8uCCLhAAInpYQxg31ZgVyiAwv2QdIw== 2024-04-25T06:49:44.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAI/Slm8uCCLhAAInpYQxg31ZgVyiAwv2QdIw=="}' 2024-04-25T06:49:44.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d72d565-b607-48cc-a487-e0ecb225826c -i td/bad-inc-map/0/new.json 2024-04-25T06:49:45.077 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:49:45.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-04-25T06:49:45.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAI/Slm8uCCLhAAInpYQxg31ZgVyiAwv2QdIw== --osd-uuid 4d72d565-b607-48cc-a487-e0ecb225826c 2024-04-25T06:49:45.164 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:45.162+0000 7fc441a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:49:45.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:45.162+0000 7fc441a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:49:45.167 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:45.166+0000 7fc441a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:49:45.168 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:45.166+0000 7fc441a597c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T06:49:45.168 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:45.166+0000 7fc441a597c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T06:49:45.168 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:45.166+0000 7fc441a597c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T06:49:45.228 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:45.226+0000 7fc441a597c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-04-25T06:49:51.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-04-25T06:49:51.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:49:51.213 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T06:49:51.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T06:49:51.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:49:51.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T06:49:51.453 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T06:49:51.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:49:51.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:49:51.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:49:51.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:49:51.483 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:51.482+0000 7f3a3b4857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:49:51.485 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:51.482+0000 7f3a3b4857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:49:51.487 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:51.486+0000 7f3a3b4857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:49:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T06:49:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:49:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T06:49:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:49:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:49:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:49:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:49:51.675 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:49:51.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:49:51.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:49:51.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:49:52.065 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:52.062+0000 7f3a3b4857c0 -1 Falling back to public interface 2024-04-25T06:49:52.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:49:52.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:49:52.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:49:52.893 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:49:52.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:49:52.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:49:53.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:49:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:49:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:49:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:49:54.190 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:49:54.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:49:54.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:49:54.355 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:49:54.354+0000 7f3a3b4857c0 -1 osd.0 0 log_to_monitors true 2024-04-25T06:49:54.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:49:55.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:49:55.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:49:55.634 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:49:55.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:49:55.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:49:55.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:49:56.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:49:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:49:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:49:57.152 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:49:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:49:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:49:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:49:57.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:49:58.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:49:58.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:49:58.572 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:49:58.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:49:58.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:49:58.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:49:59.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:00.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:00.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:00.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:50:00.041 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:50:00.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:00.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:00.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:01.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:01.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:01.467 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:50:01.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:50:01.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:01.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:01.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:02.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:02.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:02.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:50:02.883 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:50:02.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:02.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:03.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:04.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:04.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:04.354 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:50:04.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:50:04.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:04.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:04.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:05.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:05.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:05.786 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:50:05.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:50:05.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:05.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:06.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:07.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:07.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:07.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:50:07.492 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:50:07.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:07.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:08.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:09.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:09.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:09.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:50:09.251 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:50:09.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:09.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:10.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:11.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:50:11.107 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:50:11.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:11.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:11.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:12.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:12.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:12.540 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:50:12.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:50:12.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:12.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:12.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:13.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:13.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:13.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:50:13.999 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:50:13.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:14.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:14.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:15.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T06:50:15.720 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T06:50:15.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:15.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:16.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:17.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:17.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:17.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T06:50:17.931 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T06:50:17.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:17.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:50:18.410 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1989898325,v1:127.0.0.1:6801/1989898325] [v2:127.0.0.1:6802/1989898325,v1:127.0.0.1:6803/1989898325] exists,up 4d72d565-b607-48cc-a487-e0ecb225826c 2024-04-25T06:50:18.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 ' 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:50:18.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20256/$cluster-$name.asok' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:50:18.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:50:18.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-04-25T06:50:18.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:50:18.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30d111c5-af82-4960-871e-8a188dfc6584 2024-04-25T06:50:18.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 30d111c5-af82-4960-871e-8a188dfc6584' 2024-04-25T06:50:18.417 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 30d111c5-af82-4960-871e-8a188dfc6584 2024-04-25T06:50:18.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:50:18.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAq/SlmHWb1GRAA1Dh9toE22Q/KPPqizMfCkw== 2024-04-25T06:50:18.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAq/SlmHWb1GRAA1Dh9toE22Q/KPPqizMfCkw=="}' 2024-04-25T06:50:18.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30d111c5-af82-4960-871e-8a188dfc6584 -i td/bad-inc-map/1/new.json 2024-04-25T06:50:18.989 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:50:19.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-04-25T06:50:19.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAq/SlmHWb1GRAA1Dh9toE22Q/KPPqizMfCkw== --osd-uuid 30d111c5-af82-4960-871e-8a188dfc6584 2024-04-25T06:50:19.032 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:19.034+0000 7f048f2c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:19.064 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:19.066+0000 7f048f2c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:19.066 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:19.066+0000 7f048f2c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:19.066 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:19.070+0000 7f048f2c97c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T06:50:19.067 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:19.070+0000 7f048f2c97c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T06:50:19.067 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:19.070+0000 7f048f2c97c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T06:50:19.147 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:19.150+0000 7f048f2c97c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-04-25T06:50:24.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-04-25T06:50:24.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:50:24.742 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T06:50:24.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T06:50:24.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:50:25.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T06:50:25.338 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T06:50:25.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:50:25.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:50:25.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:50:25.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:50:25.370 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:25.370+0000 7f7bc1bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:25.403 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:25.406+0000 7f7bc1bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:25.404 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:25.406+0000 7f7bc1bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:25.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T06:50:25.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:50:25.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T06:50:25.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:50:25.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:50:25.780 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:50:25.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:25.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:50:25.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:25.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:25.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:25.974+0000 7f7bc1bc27c0 -1 Falling back to public interface 2024-04-25T06:50:26.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:27.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:27.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:27.225 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:50:27.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:50:27.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:27.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:27.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:28.443 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:28.446+0000 7f7bc1bc27c0 -1 osd.1 0 log_to_monitors true 2024-04-25T06:50:28.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:28.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:28.670 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:50:28.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:50:28.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:28.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:29.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:30.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:30.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:30.234 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:50:30.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:50:30.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:30.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:30.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:31.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:31.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:31.738 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:50:31.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:50:31.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:31.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:32.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:33.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:33.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:33.180 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:50:33.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:50:33.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:33.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:33.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:34.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:34.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:34.592 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:50:34.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:50:34.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:34.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:35.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:36.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:36.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:36.042 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:50:36.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:50:36.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:36.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:36.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:37.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:37.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:37.498 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:50:37.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:50:37.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:37.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:37.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:38.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:38.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:38.917 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:50:38.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:50:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:38.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:39.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:40.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:40.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:40.420 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:50:40.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:50:40.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:40.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:41.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:42.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:42.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:42.146 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:50:42.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:50:42.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:42.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:42.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:43.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:43.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:43.932 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:50:43.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:50:43.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:43.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:44.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:45.811 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:50:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:50:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:46.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:47.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:47.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:47.566 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:50:47.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:50:47.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:47.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:48.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:49.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:49.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:49.336 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:50:49.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:50:49.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:49.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:50.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:50:51.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:50:51.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:51.190 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T06:50:51.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T06:50:51.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:51.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:50:51.650 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2415180151,v1:127.0.0.1:6811/2415180151] [v2:127.0.0.1:6812/2415180151,v1:127.0.0.1:6813/2415180151] exists,up 30d111c5-af82-4960-871e-8a188dfc6584 2024-04-25T06:50:51.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:50:51.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 ' 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:50:51.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20256/$cluster-$name.asok' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T06:50:51.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T06:50:51.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:50:51.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:50:51.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:50:51.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:50:51.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:50:51.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:50:51.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-04-25T06:50:51.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:50:51.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bcc8145a-aa54-49f5-9f46-d72f828061fc 2024-04-25T06:50:51.666 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 bcc8145a-aa54-49f5-9f46-d72f828061fc 2024-04-25T06:50:51.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bcc8145a-aa54-49f5-9f46-d72f828061fc' 2024-04-25T06:50:51.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:50:51.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBL/SlmFvvBKBAAbVwNYt4fNIk6z5PTX97iVw== 2024-04-25T06:50:51.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBL/SlmFvvBKBAAbVwNYt4fNIk6z5PTX97iVw=="}' 2024-04-25T06:50:51.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bcc8145a-aa54-49f5-9f46-d72f828061fc -i td/bad-inc-map/2/new.json 2024-04-25T06:50:52.194 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:50:52.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-04-25T06:50:52.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBL/SlmFvvBKBAAbVwNYt4fNIk6z5PTX97iVw== --osd-uuid bcc8145a-aa54-49f5-9f46-d72f828061fc 2024-04-25T06:50:52.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:52.248+0000 7ff02e6577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:52.266 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:52.264+0000 7ff02e6577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:52.268 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:52.268+0000 7ff02e6577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:52.268 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:52.268+0000 7ff02e6577c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T06:50:52.268 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:52.268+0000 7ff02e6577c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T06:50:52.268 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:52.268+0000 7ff02e6577c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T06:50:52.359 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:52.360+0000 7ff02e6577c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-04-25T06:50:58.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-04-25T06:50:58.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:50:58.162 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T06:50:58.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T06:50:58.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:50:58.718 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T06:50:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T06:50:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=86d2b4f1-3acb-46d8-b6b6-01c92eed91a3 --auth-supported=none --mon-host=127.0.0.1:56063 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20256/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:50:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:50:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:50:58.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:50:58.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:58.748+0000 7f0ecff647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:58.764 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:58.764+0000 7f0ecff647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:58.765 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:58.764+0000 7f0ecff647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:50:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T06:50:59.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:50:59.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T06:50:59.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:50:59.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:50:59.159 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:50:59.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:50:59.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:50:59.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:50:59.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:50:59.325 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:50:59.324+0000 7f0ecff647c0 -1 Falling back to public interface 2024-04-25T06:50:59.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:00.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:00.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:00.622 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:51:00.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:51:00.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:00.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:01.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:01.896 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:01.895+0000 7f0ecff647c0 -1 osd.2 0 log_to_monitors true 2024-04-25T06:51:02.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:02.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:02.021 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:51:02.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:51:02.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:02.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:02.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:03.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:03.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:03.567 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:51:03.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:51:03.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:03.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:05.086 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:51:05.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:51:05.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:05.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:05.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:06.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:06.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:06.505 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:51:06.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:51:06.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:06.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:07.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:08.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:08.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:08.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:51:08.047 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:51:08.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:08.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:08.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:09.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:09.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:09.509 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:51:09.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:51:09.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:09.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:11.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:11.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:11.096 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:51:11.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:51:11.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:11.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:11.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:12.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:12.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:12.496 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:51:12.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:51:12.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:12.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:13.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:14.001 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:51:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:51:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:14.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:15.915 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:51:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:51:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:16.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:17.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:17.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:17.763 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:51:17.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:51:17.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:17.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:18.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:19.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:19.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:19.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:51:19.405 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:51:19.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:19.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:20.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:21.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:21.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:21.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:51:21.211 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:51:21.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:21.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:21.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:22.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:22.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:22.941 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:51:22.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:51:22.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:22.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:23.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:24.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:24.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:24.366 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T06:51:24.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T06:51:24.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:24.831 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3168835956,v1:127.0.0.1:6819/3168835956] [v2:127.0.0.1:6820/3168835956,v1:127.0.0.1:6821/3168835956] exists,up bcc8145a-aa54-49f5-9f46-d72f828061fc 2024-04-25T06:51:24.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:51:24.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:51:24.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:51:24.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-04-25T06:51:25.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-04-25T06:51:25.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-04-25T06:51:25.914 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T06:51:25.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T06:51:26.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-04-25T06:51:27.966 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 1 2024-04-25T06:51:27.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-04-25T06:51:29.048 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 2 2024-04-25T06:51:29.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-04-25T06:51:34.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T06:51:34.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-04-25T06:51:34.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:51:34.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T06:51:34.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:51:34.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:51:34.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T06:51:34.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:51:34.076 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:51:34.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:34.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:51:34.489 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1989898325,v1:127.0.0.1:6801/1989898325] [v2:127.0.0.1:6802/1989898325,v1:127.0.0.1:6803/1989898325] exists,up 4d72d565-b607-48cc-a487-e0ecb225826c 2024-04-25T06:51:34.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:51:34.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:51:34.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:51:34.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T06:51:34.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-04-25T06:51:34.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:51:34.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T06:51:34.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:51:34.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:51:34.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T06:51:34.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:51:34.492 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:51:34.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:34.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:51:34.906 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2415180151,v1:127.0.0.1:6811/2415180151] [v2:127.0.0.1:6812/2415180151,v1:127.0.0.1:6813/2415180151] exists,up 30d111c5-af82-4960-871e-8a188dfc6584 2024-04-25T06:51:34.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:51:34.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:51:34.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:51:34.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T06:51:34.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-04-25T06:51:34.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:51:34.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T06:51:34.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:51:34.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:51:34.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T06:51:34.908 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:51:34.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:51:34.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:34.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:51:35.348 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3168835956,v1:127.0.0.1:6819/3168835956] [v2:127.0.0.1:6820/3168835956,v1:127.0.0.1:6821/3168835956] exists,up bcc8145a-aa54-49f5-9f46-d72f828061fc 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:25.782+0000 7f0ec0f08640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:26.118+0000 7f0ec0f08640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:26.486+0000 7f0ec0f08640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:26.862+0000 7f0ec0f08640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:27.734+0000 7f0ebef04640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:28.146+0000 7f0ebef04640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:28.234+0000 7f0ebef04640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.373 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:28.318+0000 7f0ec0f08640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.374 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:28.698+0000 7f0ec0f08640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.374 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:29.094+0000 7f0ec0f08640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.374 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:29.458+0000 7f0ec0f08640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.374 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:29.870+0000 7f0ec0f08640 -1 osd.2 27 handle_osd_map injecting map crc failure 2024-04-25T06:51:35.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-04-25T06:51:35.374 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:25.782+0000 7f0ec0f08640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-04-25T06:51:35.374 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:26.122+0000 7f0ec0f08640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-04-25T06:51:35.374 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:26.486+0000 7f0ec0f08640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:26.866+0000 7f0ec0f08640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:27.738+0000 7f0ebef04640 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:28.146+0000 7f0ebef04640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:28.234+0000 7f0ebef04640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:28.318+0000 7f0ec0f08640 10 osd.2 22 handle_osd_map bailing because last < start (23<24) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:28.702+0000 7f0ec0f08640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:29.094+0000 7f0ec0f08640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:29.458+0000 7f0ec0f08640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T06:51:29.870+0000 7f0ec0f08640 10 osd.2 27 handle_osd_map bailing because last < start (27<28) 2024-04-25T06:51:35.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-04-25T06:51:35.376 INFO:tasks.workunit.client.0.smithi088.stdout:success 2024-04-25T06:51:35.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-04-25T06:51:35.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-04-25T06:51:35.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-04-25T06:51:35.934 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' does not exist 2024-04-25T06:51:35.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-04-25T06:51:35.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:51:35.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:51:35.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:51:35.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:51:35.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:51:41.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:51:41.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:51:41.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T06:51:41.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T06:51:41.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T06:51:41.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T06:51:41.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T06:51:41.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T06:51:41.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:51:41.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T06:51:41.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T06:51:41.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:51:41.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:51:41.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T06:51:41.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T06:51:41.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T06:51:41.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:41.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:51:41.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20256 2024-04-25T06:51:41.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/bad-inc-map 0 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:51:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:51:41.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:51:41.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T06:51:41.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T06:51:41.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T06:51:41.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T06:51:41.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T06:51:41.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T06:51:41.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:51:41.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T06:51:41.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T06:51:41.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:51:41.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:51:41.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T06:51:41.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T06:51:41.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T06:51:41.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:41.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20256 2024-04-25T06:51:41.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20256 2024-04-25T06:51:41.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T06:51:41.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T06:51:41.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T06:51:41.310 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T06:51:41.310 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T06:51:41.362 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-04-25T06:51:41.362 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-04-25T06:51:41.417 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T06:51:41.424 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T06:51:41.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T06:51:41.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T06:51:41.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/divergent-priors 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-04-25T06:51:41.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-04-25T06:51:41.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-04-25T06:51:41.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-04-25T06:51:41.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-04-25T06:51:41.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-04-25T06:51:41.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-04-25T06:51:41.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-04-25T06:51:41.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-04-25T06:51:41.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-04-25T06:51:41.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-04-25T06:51:41.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-04-25T06:51:41.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-04-25T06:51:41.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none ' 2024-04-25T06:51:41.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T06:51:41.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T06:51:41.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-04-25T06:51:41.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:51:41.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-04-25T06:51:41.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_divergent_2 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_divergent_3 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_divergent_ec' 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T06:51:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T06:51:41.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T06:51:41.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:51:41.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:51:41.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:51:41.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:51:41.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:51:41.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:51:41.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T06:51:41.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T06:51:41.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T06:51:41.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T06:51:41.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T06:51:41.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T06:51:41.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:51:41.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T06:51:41.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T06:51:41.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:51:41.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:51:41.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T06:51:41.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T06:51:41.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T06:51:41.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:41.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:51:41.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T06:51:41.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T06:51:41.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T06:51:41.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T06:51:41.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T06:51:41.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:41.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:51:41.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26017 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-04-25T06:51:41.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-04-25T06:51:41.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-04-25T06:51:41.448 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-04-25T06:51:41.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T06:51:41.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T06:51:41.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T06:51:41.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:51:41.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:51:41.896 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:51:41.896 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:41.896 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:51:41.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:51:41.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T06:51:42.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T06:51:42.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T06:51:42.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T06:51:42.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T06:51:42.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T06:51:42.098 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T06:51:42.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T06:51:42.098 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T06:51:42.098 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T06:51:42.098 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:51:42.098 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:42.098 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:51:42.099 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-mon.a.asok 2024-04-25T06:51:42.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T06:51:42.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26017/ceph-mon.a.asok config get fsid 2024-04-25T06:51:42.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T06:51:42.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T06:51:42.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T06:51:42.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T06:51:42.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T06:51:42.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T06:51:42.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T06:51:42.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T06:51:42.215 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:51:42.215 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:42.215 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:51:42.216 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-mon.a.asok 2024-04-25T06:51:42.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T06:51:42.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26017/ceph-mon.a.asok config get mon_host 2024-04-25T06:51:42.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-04-25T06:51:42.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T06:51:42.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T06:51:42.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T06:51:42.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T06:51:42.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T06:51:42.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T06:51:43.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T06:51:43.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:51:43.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:51:43.230 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:51:43.230 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:43.230 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:51:43.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:51:43.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T06:51:43.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:51:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:51:43.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:51:43.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5ca31b24-37e3-4416-a84f-76733a34590b 2024-04-25T06:51:43.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5ca31b24-37e3-4416-a84f-76733a34590b' 2024-04-25T06:51:43.272 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 5ca31b24-37e3-4416-a84f-76733a34590b 2024-04-25T06:51:43.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:51:43.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB//SlmxTiZERAAU1UqHR0jrXjHAGwqLXwfgw== 2024-04-25T06:51:43.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB//SlmxTiZERAAU1UqHR0jrXjHAGwqLXwfgw=="}' 2024-04-25T06:51:43.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5ca31b24-37e3-4416-a84f-76733a34590b -i td/divergent-priors/0/new.json 2024-04-25T06:51:43.593 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:51:43.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T06:51:43.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB//SlmxTiZERAAU1UqHR0jrXjHAGwqLXwfgw== --osd-uuid 5ca31b24-37e3-4416-a84f-76733a34590b 2024-04-25T06:51:43.643 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:43.638+0000 7fbc868867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:51:43.662 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:43.658+0000 7fbc868867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:51:43.663 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:43.658+0000 7fbc868867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:51:43.663 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:43.658+0000 7fbc868867c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T06:51:43.663 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:43.658+0000 7fbc868867c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T06:51:43.663 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:43.658+0000 7fbc868867c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T06:51:43.702 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:43.698+0000 7fbc868867c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T06:51:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T06:51:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:51:49.601 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T06:51:49.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T06:51:49.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:51:50.183 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T06:51:50.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T06:51:50.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:51:50.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:51:50.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:51:50.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:51:50.223 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:50.218+0000 7f915c7d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:51:50.258 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:50.254+0000 7f915c7d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:51:50.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:50.254+0000 7f915c7d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:51:50.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T06:51:50.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:51:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T06:51:50.640 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:51:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:51:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:51:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:51:50.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:50.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:51:50.821 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:50.822+0000 7f915c7d57c0 -1 Falling back to public interface 2024-04-25T06:51:51.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:52.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:52.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:52.118 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:51:52.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:51:52.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:52.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:51:52.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:53.318 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:51:53.318+0000 7f915c7d57c0 -1 osd.0 0 log_to_monitors true 2024-04-25T06:51:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:51:53.535 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:51:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:51:54.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:55.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:55.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:55.014 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:51:55.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:51:55.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:55.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:51:55.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:56.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:56.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:56.468 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:51:56.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:51:56.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:56.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:51:56.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:57.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:57.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:57.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:51:57.998 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:51:57.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:57.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:51:58.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:51:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:51:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:51:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:51:59.430 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:51:59.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:51:59.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:51:59.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:00.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:00.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:00.872 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:52:00.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:52:00.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:00.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:01.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:02.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:02.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:02.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:52:02.313 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:52:02.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:02.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:02.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:03.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:03.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:03.756 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:52:03.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:52:03.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:03.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:04.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:05.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:05.441 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:52:05.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:05.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:52:05.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:05.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:05.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:06.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:06.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:06.938 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:52:06.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:52:06.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:06.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:07.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:08.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:08.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:08.381 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:52:08.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:52:08.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:08.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:09.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:10.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:10.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:10.010 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:52:10.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:52:10.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:10.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:11.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:11.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:11.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:52:11.819 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:52:11.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:11.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:12.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:13.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:13.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:13.305 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:52:13.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:52:13.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:13.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:13.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:14.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:14.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:14.987 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T06:52:14.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T06:52:14.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:14.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:52:15.505 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/411842594,v1:127.0.0.1:6803/411842594] [v2:127.0.0.1:6804/411842594,v1:127.0.0.1:6805/411842594] exists,up 5ca31b24-37e3-4416-a84f-76733a34590b 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:52:15.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:52:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:52:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T06:52:15.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:52:15.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58ee56f4-4e3c-499a-8a46-1f4513ef1de8 2024-04-25T06:52:15.510 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 58ee56f4-4e3c-499a-8a46-1f4513ef1de8 2024-04-25T06:52:15.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 58ee56f4-4e3c-499a-8a46-1f4513ef1de8' 2024-04-25T06:52:15.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:52:15.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCf/Slm2My1HxAAitXjfo6hBBXd5BQofdFNDw== 2024-04-25T06:52:15.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCf/Slm2My1HxAAitXjfo6hBBXd5BQofdFNDw=="}' 2024-04-25T06:52:15.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58ee56f4-4e3c-499a-8a46-1f4513ef1de8 -i td/divergent-priors/1/new.json 2024-04-25T06:52:16.143 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:52:16.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T06:52:16.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCf/Slm2My1HxAAitXjfo6hBBXd5BQofdFNDw== --osd-uuid 58ee56f4-4e3c-499a-8a46-1f4513ef1de8 2024-04-25T06:52:16.207 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:16.205+0000 7f5bb00c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:16.209 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:16.209+0000 7f5bb00c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:16.210 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:16.209+0000 7f5bb00c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:16.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:16.209+0000 7f5bb00c47c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T06:52:16.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:16.209+0000 7f5bb00c47c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T06:52:16.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:16.209+0000 7f5bb00c47c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T06:52:16.299 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:16.297+0000 7f5bb00c47c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T06:52:22.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T06:52:22.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:52:22.184 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T06:52:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T06:52:22.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:52:22.814 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T06:52:22.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T06:52:22.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:52:22.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:52:22.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:52:22.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:52:22.855 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:22.853+0000 7f8ef25637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:22.858 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:22.857+0000 7f8ef25637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:22.859 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:22.857+0000 7f8ef25637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T06:52:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:52:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T06:52:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:52:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:52:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:23.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:52:23.267 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:52:23.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:23.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:23.434 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:23.433+0000 7f8ef25637c0 -1 Falling back to public interface 2024-04-25T06:52:23.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:24.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:24.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:24.678 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:52:24.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:52:24.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:24.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:25.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:26.021 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:26.021+0000 7f8ef25637c0 -1 osd.1 0 log_to_monitors true 2024-04-25T06:52:26.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:26.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:26.155 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:52:26.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:52:26.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:26.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:26.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:27.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:27.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:27.761 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:52:27.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:52:27.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:27.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:28.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:29.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:29.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:29.204 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:52:29.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:52:29.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:29.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:29.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:30.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:30.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:30.651 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:52:30.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:52:30.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:30.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:31.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:32.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:32.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:32.234 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:52:32.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:52:32.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:32.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:32.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:33.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:33.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:33.737 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:52:33.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:52:33.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:33.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:34.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:35.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:35.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:35.215 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:52:35.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:52:35.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:35.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:35.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:36.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:36.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:36.672 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:52:36.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:52:36.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:36.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:37.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:38.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:38.189 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:52:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:52:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:38.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:38.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:39.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:39.953 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:52:39.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:39.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:52:39.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:39.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:41.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:41.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:41.970 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:52:41.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:52:41.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:41.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:42.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:43.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:43.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:43.701 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:52:43.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:52:43.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:43.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:44.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:45.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:45.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:45.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:52:45.750 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:52:45.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:45.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:47.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:48.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:48.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:48.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:52:48.108 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:52:48.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:48.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:52:48.569 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/228050032,v1:127.0.0.1:6811/228050032] [v2:127.0.0.1:6812/228050032,v1:127.0.0.1:6813/228050032] exists,up 58ee56f4-4e3c-499a-8a46-1f4513ef1de8 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:52:48.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:52:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:52:48.574 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:52:48.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:52:48.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:52:48.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:52:48.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:52:48.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:52:48.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:52:48.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:52:48.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9082884f-9bb9-4ce7-8e79-15c15c6660a6 2024-04-25T06:52:48.577 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 9082884f-9bb9-4ce7-8e79-15c15c6660a6 2024-04-25T06:52:48.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9082884f-9bb9-4ce7-8e79-15c15c6660a6' 2024-04-25T06:52:48.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDA/Slm/zCyIxAAZkgsL3kfZ+P2HGGYv/+xUA== 2024-04-25T06:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDA/Slm/zCyIxAAZkgsL3kfZ+P2HGGYv/+xUA=="}' 2024-04-25T06:52:48.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9082884f-9bb9-4ce7-8e79-15c15c6660a6 -i td/divergent-priors/2/new.json 2024-04-25T06:52:49.078 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:52:49.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T06:52:49.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDA/Slm/zCyIxAAZkgsL3kfZ+P2HGGYv/+xUA== --osd-uuid 9082884f-9bb9-4ce7-8e79-15c15c6660a6 2024-04-25T06:52:49.131 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:49.128+0000 7f8995de17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:49.133 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:49.132+0000 7f8995de17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:49.135 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:49.132+0000 7f8995de17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:49.135 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:49.132+0000 7f8995de17c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T06:52:49.135 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:49.132+0000 7f8995de17c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T06:52:49.135 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:49.132+0000 7f8995de17c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T06:52:49.177 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:49.176+0000 7f8995de17c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T06:52:55.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T06:52:55.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:52:55.355 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T06:52:55.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T06:52:55.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:52:55.949 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T06:52:55.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T06:52:55.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:52:55.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:52:55.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:52:55.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:52:55.988 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:55.984+0000 7f8797e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:55.990 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:55.988+0000 7f8797e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:55.991 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:55.988+0000 7f8797e8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:52:56.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T06:52:56.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:52:56.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T06:52:56.403 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:52:56.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:52:56.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:52:56.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:56.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:52:56.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:52:56.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:56.582 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:56.580+0000 7f8797e8f7c0 -1 Falling back to public interface 2024-04-25T06:52:56.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:57.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:57.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:57.978 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:52:57.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:52:57.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:57.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:52:58.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:52:59.140 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:52:59.136+0000 7f8797e8f7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T06:52:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:52:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:52:59.429 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:52:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:52:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:52:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:52:59.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:00.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:00.894 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:53:00.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:53:00.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:00.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:01.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:02.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:02.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:02.405 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:53:02.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:53:02.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:02.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:03.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:04.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:04.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:04.054 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:53:04.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:53:04.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:04.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:05.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:05.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:05.626 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:53:05.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:53:05.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:05.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:06.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:07.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:07.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:07.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:53:07.048 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:53:07.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:07.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:07.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:08.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:08.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:08.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:53:08.494 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:53:08.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:08.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:09.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:10.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:10.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:10.049 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:53:10.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:53:10.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:10.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:10.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:11.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:11.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:11.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:53:11.630 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:53:11.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:11.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:12.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:13.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:13.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:13.575 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:53:13.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:53:13.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:13.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:14.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:15.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:15.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:15.395 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:53:15.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:53:15.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:15.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:17.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:17.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:17.481 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:53:17.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:53:17.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:17.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:18.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:19.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:19.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:19.318 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:53:19.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:53:19.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:19.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:20.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:21.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:21.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:21.141 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:53:21.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:53:21.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:21.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:21.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:53:22.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:53:22.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:53:22.951 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T06:53:22.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T06:53:22.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:53:22.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:53:23.406 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4190646884,v1:127.0.0.1:6819/4190646884] [v2:127.0.0.1:6820/4190646884,v1:127.0.0.1:6821/4190646884] exists,up 9082884f-9bb9-4ce7-8e79-15c15c6660a6 2024-04-25T06:53:23.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:53:23.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:53:23.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:53:23.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-04-25T06:53:24.095 INFO:tasks.workunit.client.0.smithi088.stderr:noout is set 2024-04-25T06:53:24.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-04-25T06:53:24.891 INFO:tasks.workunit.client.0.smithi088.stderr:noin is set 2024-04-25T06:53:24.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-04-25T06:53:25.744 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T06:53:25.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-04-25T06:53:25.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T06:53:26.419 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T06:53:26.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T06:53:27.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-04-25T06:53:28.315 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 3 2024-04-25T06:53:28.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-04-25T06:53:28.992 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 2 2024-04-25T06:53:29.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-04-25T06:53:29.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:53:29.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:53:29.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:53:29.440 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:53:29.441 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:53:29.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:53:29.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:29.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:53:29.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-04-25T06:53:29.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-04-25T06:53:29.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-04-25T06:53:29.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:29.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:53:29.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T06:53:29.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T06:53:29.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-04-25T06:53:29.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:29.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:53:29.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509447 2024-04-25T06:53:29.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509447 2024-04-25T06:53:29.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509447' 2024-04-25T06:53:29.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:29.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-04-25T06:53:29.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:29.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:53:29.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-04-25T06:53:29.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:29.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-04-25T06:53:29.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-25T06:53:29.926 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836502 2024-04-25T06:53:29.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:53:30.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836502 2024-04-25T06:53:30.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:30.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T06:53:30.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:30.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T06:53:30.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T06:53:30.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:30.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T06:53:30.340 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672975 2024-04-25T06:53:30.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T06:53:30.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T06:53:30.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672975 2024-04-25T06:53:30.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:30.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:30.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509447 2024-04-25T06:53:30.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:53:30.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:30.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509447 2024-04-25T06:53:30.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509447 2024-04-25T06:53:30.749 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509447 2024-04-25T06:53:30.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-04-25T06:53:30.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:53:31.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509445 -lt 64424509447 2024-04-25T06:53:31.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T06:53:32.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T06:53:32.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:53:32.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509447 -lt 64424509447 2024-04-25T06:53:32.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-04-25T06:53:32.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T06:53:32.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T06:53:32.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T06:53:32.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T06:53:32.687 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T06:53:32.687 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T06:53:32.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T06:53:32.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T06:53:32.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T06:53:32.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T06:53:32.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T06:53:32.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T06:53:32.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T06:53:32.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:53:32.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:53:33.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:53:33.252 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:53:33.253 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:53:33.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:53:33.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:33.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:53:33.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836505 2024-04-25T06:53:33.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836505 2024-04-25T06:53:33.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505' 2024-04-25T06:53:33.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:33.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:53:33.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-04-25T06:53:33.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-04-25T06:53:33.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-04-25T06:53:33.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:33.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:53:33.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-04-25T06:53:33.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-04-25T06:53:33.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509450' 2024-04-25T06:53:33.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:33.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:33.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836505 2024-04-25T06:53:33.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:53:33.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-04-25T06:53:33.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836505 2024-04-25T06:53:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-04-25T06:53:33.713 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836505 2024-04-25T06:53:33.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:53:34.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836505 2024-04-25T06:53:34.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:34.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-04-25T06:53:34.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:34.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T06:53:34.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-04-25T06:53:34.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:34.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-04-25T06:53:34.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-25T06:53:34.178 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672977 2024-04-25T06:53:34.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T06:53:34.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672977 2024-04-25T06:53:34.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:34.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-04-25T06:53:34.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:34.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:53:34.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-04-25T06:53:34.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:34.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-04-25T06:53:34.636 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509450 2024-04-25T06:53:34.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-04-25T06:53:34.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:53:35.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509450 2024-04-25T06:53:35.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T06:53:35.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:53:35.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:53:35.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T06:53:35.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:53:35.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:53:35.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:53:35.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:53:35.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:53:35.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:53:35.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:53:36.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T06:53:36.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:53:36.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:53:36.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:53:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T06:53:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T06:53:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T06:53:36.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-04-25T06:53:36.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-04-25T06:53:36.978 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:53:36.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-04-25T06:53:36.994 INFO:tasks.workunit.client.0.smithi088.stdout:primary and soon to be divergent is 1 2024-04-25T06:53:36.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-04-25T06:53:36.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-04-25T06:53:37.394 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:53:37.395 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T06:53:27.313124+0000 0'0 29:21 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:41:32.581066+0000 0 0 2024-04-25T06:53:37.395 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:53:37.395 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:53:37.395 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-04-25T06:53:37.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-04-25T06:53:37.413 INFO:tasks.workunit.client.0.smithi088.stdout:writing initial objects 2024-04-25T06:53:37.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-04-25T06:53:37.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-04-25T06:53:37.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:37.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-04-25T06:53:37.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:37.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-04-25T06:53:37.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:37.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-04-25T06:53:37.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:37.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-04-25T06:53:38.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:38.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-04-25T06:53:38.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:38.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-04-25T06:53:38.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:38.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-04-25T06:53:38.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:38.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-04-25T06:53:38.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:38.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-04-25T06:53:38.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:38.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-04-25T06:53:38.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:38.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-04-25T06:53:39.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:39.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-04-25T06:53:39.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:39.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-04-25T06:53:39.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:39.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-04-25T06:53:39.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:39.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-04-25T06:53:39.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:39.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-04-25T06:53:39.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:39.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-04-25T06:53:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-04-25T06:53:40.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:40.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-04-25T06:53:40.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:40.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-04-25T06:53:40.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:40.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-04-25T06:53:40.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:40.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-04-25T06:53:40.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:40.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-04-25T06:53:41.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:41.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-04-25T06:53:41.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:41.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-04-25T06:53:41.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:41.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-04-25T06:53:41.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:41.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-04-25T06:53:41.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:41.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-04-25T06:53:41.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:41.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-04-25T06:53:42.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:42.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-04-25T06:53:42.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:42.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-04-25T06:53:42.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:42.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-04-25T06:53:42.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:42.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-04-25T06:53:42.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:42.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-04-25T06:53:42.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:42.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-04-25T06:53:42.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:42.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-04-25T06:53:43.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:43.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-04-25T06:53:43.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:43.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-04-25T06:53:43.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:43.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-04-25T06:53:43.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:43.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-04-25T06:53:43.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:43.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-04-25T06:53:43.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:43.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-04-25T06:53:44.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:44.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-04-25T06:53:44.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:44.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-04-25T06:53:44.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:44.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-04-25T06:53:44.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:44.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-04-25T06:53:44.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:44.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-04-25T06:53:44.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:44.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-04-25T06:53:45.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:45.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-04-25T06:53:45.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:45.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-04-25T06:53:45.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:45.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-04-25T06:53:45.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:45.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-04-25T06:53:45.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:45.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-04-25T06:53:45.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:45.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-04-25T06:53:46.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:46.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-04-25T06:53:46.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:46.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-04-25T06:53:46.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:46.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-04-25T06:53:46.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:46.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-04-25T06:53:46.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:46.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-04-25T06:53:46.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:46.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-04-25T06:53:47.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:47.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-04-25T06:53:47.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:47.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-04-25T06:53:47.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:47.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-04-25T06:53:47.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:47.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-04-25T06:53:47.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:47.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-04-25T06:53:47.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:47.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-04-25T06:53:48.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:48.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-04-25T06:53:48.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:48.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-04-25T06:53:48.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:48.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-04-25T06:53:48.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:48.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-04-25T06:53:48.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:48.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-04-25T06:53:48.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:48.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-04-25T06:53:49.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:49.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-04-25T06:53:49.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:49.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-04-25T06:53:49.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:49.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-04-25T06:53:49.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:49.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-04-25T06:53:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-04-25T06:53:49.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:49.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-04-25T06:53:50.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:50.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-04-25T06:53:50.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:50.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-04-25T06:53:50.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:50.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-04-25T06:53:50.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:50.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-04-25T06:53:50.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:50.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-04-25T06:53:50.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:50.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-04-25T06:53:51.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:51.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-04-25T06:53:51.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:51.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-04-25T06:53:51.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:51.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-04-25T06:53:51.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:51.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-04-25T06:53:51.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:51.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-04-25T06:53:51.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:51.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-04-25T06:53:51.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:51.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-04-25T06:53:52.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:52.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-04-25T06:53:52.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:52.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-04-25T06:53:52.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:52.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-04-25T06:53:52.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:52.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-04-25T06:53:52.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:52.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-04-25T06:53:52.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:52.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-04-25T06:53:53.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:53.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-04-25T06:53:53.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:53.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-04-25T06:53:53.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T06:53:53.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-04-25T06:53:53.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T06:53:53.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-04-25T06:53:53.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T06:53:53.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T06:53:53.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T06:53:53.665 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T06:53:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T06:53:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T06:53:53.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T06:53:53.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T06:53:53.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T06:53:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T06:53:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T06:53:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T06:53:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T06:53:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:53:53.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:53:54.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:53:54.186 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:53:54.186 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:53:54.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:53:54.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:54.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:53:54.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T06:53:54.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T06:53:54.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T06:53:54.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:54.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:53:54.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672984 2024-04-25T06:53:54.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672984 2024-04-25T06:53:54.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672984' 2024-04-25T06:53:54.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:53:54.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:53:54.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-04-25T06:53:54.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-04-25T06:53:54.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672984 2-64424509456' 2024-04-25T06:53:54.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:54.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T06:53:54.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:54.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:53:54.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T06:53:54.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:54.671 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836511 2024-04-25T06:53:54.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T06:53:54.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T06:53:54.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:53:55.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836511 2024-04-25T06:53:55.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T06:53:56.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T06:53:56.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:53:56.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836511 2024-04-25T06:53:56.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:56.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672984 2024-04-25T06:53:56.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:56.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T06:53:56.549 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672984 2024-04-25T06:53:56.549 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:56.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672984 2024-04-25T06:53:56.551 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672984 2024-04-25T06:53:56.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-04-25T06:53:56.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T06:53:57.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672984 -lt 42949672984 2024-04-25T06:53:57.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:53:57.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-04-25T06:53:57.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:53:57.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:53:57.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-04-25T06:53:57.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:53:57.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-04-25T06:53:57.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-25T06:53:57.024 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509456 2024-04-25T06:53:57.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:53:57.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509456 2024-04-25T06:53:57.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T06:53:57.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:53:57.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:53:57.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T06:53:57.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:53:57.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:53:57.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:53:57.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:53:57.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:53:57.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:53:57.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:53:58.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T06:53:58.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:53:58.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:53:58.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:53:58.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T06:53:58.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T06:53:58.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T06:53:58.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-04-25T06:53:58.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T06:53:58.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T06:53:58.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T06:53:58.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T06:53:59.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-04-25T06:53:59.423 INFO:tasks.workunit.client.0.smithi088.stdout:blackholing osds 0 2 2024-04-25T06:53:59.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-04-25T06:53:59.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-04-25T06:53:59.846 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:53:59.846 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T06:53:27.313124+0000 29'100 29:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:41:32.581066+0000 0 0 2024-04-25T06:53:59.846 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:53:59.846 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:53:59.846 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:53:59.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-04-25T06:53:59.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-04-25T06:53:59.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T06:53:59.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T06:53:59.863 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:53:59.863 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:53:59.864 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:53:59.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.0.asok 2024-04-25T06:53:59.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-04-25T06:53:59.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T06:53:59.958 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T06:53:59.958 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T06:53:59.958 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T06:53:59.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-04-25T06:53:59.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-04-25T06:53:59.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T06:53:59.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T06:53:59.971 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:53:59.971 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:53:59.971 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:53:59.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.2.asok 2024-04-25T06:53:59.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-04-25T06:53:59.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T06:54:00.080 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T06:54:00.080 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T06:54:00.080 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T06:54:00.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-04-25T06:54:00.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-04-25T06:54:00.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-04-25T06:54:00.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-04-25T06:54:00.094 INFO:tasks.workunit.client.0.smithi088.stdout:writing divergent object 2024-04-25T06:54:00.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-04-25T06:54:00.095 INFO:tasks.workunit.client.0.smithi088.stdout:create missing divergent object 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=existing_99 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T06:54:00.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/divergent-priors 2024-04-25T06:54:00.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T06:54:00.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T06:54:00.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T06:54:00.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test existing_99 2024-04-25T06:54:00.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-04-25T06:54:00.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-04-25T06:54:00.098 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-04-25T06:54:00.098 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T06:54:00.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-25T06:54:00.559 INFO:tasks.workunit.client.0.smithi088.stderr:0 2024-04-25T06:54:00.559 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:54:00.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-04-25T06:54:00.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0' '2') 2024-04-25T06:54:00.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/divergent-priors/1/type 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T06:54:00.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T06:54:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T06:54:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T06:54:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T06:54:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:54:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:54:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:54:00.562 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.1.asok 2024-04-25T06:54:00.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.26017/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T06:54:00.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T06:54:00.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T06:54:00.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T06:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T06:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T06:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T06:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.1.asok 2024-04-25T06:54:00.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T06:54:00.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.1.asok injectdataerr test existing_99 2024-04-25T06:54:00.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-04-25T06:54:00.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-04-25T06:54:10.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-04-25T06:54:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:rados: no process found 2024-04-25T06:54:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-04-25T06:54:10.808 INFO:tasks.workunit.client.0.smithi088.stdout:killing all the osds 2024-04-25T06:54:10.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-04-25T06:54:11.216 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:54:11.216 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-04-25T06:54:00.949242+0000 29'101 29:130 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:41:32.581066+0000 0 0 2024-04-25T06:54:11.216 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:54:11.216 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:54:11.216 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:54:11.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-04-25T06:54:11.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:54:11.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:54:11.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:54:11.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:54:11.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:54:11.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:54:11.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T06:54:11.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-04-25T06:54:11.855 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T06:54:11.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T06:54:11.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-04-25T06:54:12.434 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T06:54:12.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T06:54:12.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-04-25T06:54:12.970 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already down. 2024-04-25T06:54:12.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-04-25T06:54:12.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-04-25T06:54:13.525 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T06:54:13.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-04-25T06:54:13.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-04-25T06:54:14.109 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already out. 2024-04-25T06:54:14.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-04-25T06:54:14.127 INFO:tasks.workunit.client.0.smithi088.stdout:bringing up non_divergent 0 2 2024-04-25T06:54:14.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-04-25T06:54:14.525 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:54:14.525 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-04-25T06:54:00.949242+0000 29'101 29:130 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:41:32.581066+0000 0 0 2024-04-25T06:54:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:54:14.525 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:54:14.526 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:54:14.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-04-25T06:54:14.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-04-25T06:54:14.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:54:14.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T06:54:14.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:54:14.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:54:14.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:54:14.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T06:54:14.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T06:54:14.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T06:54:14.546 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T06:54:14.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T06:54:14.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:54:14.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T06:54:14.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T06:54:14.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T06:54:14.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T06:54:14.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T06:54:14.585 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:14.583+0000 7f2e99c057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:54:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:14.583+0000 7f2e99c057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:54:14.588 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:14.587+0000 7f2e99c057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:54:14.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T06:54:14.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:54:14.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T06:54:14.996 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:54:14.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:54:14.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:54:14.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:54:14.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:54:14.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:54:14.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:54:15.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:15.163+0000 7f2e99c057c0 -1 Falling back to public interface 2024-04-25T06:54:15.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:54:16.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:54:16.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:54:16.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:54:16.422 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:54:16.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:54:16.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:54:16.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:54:17.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:54:17.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:54:17.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:54:17.863 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:54:17.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:54:17.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:54:17.951 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:17.947+0000 7f2e99c057c0 -1 osd.0 29 log_to_monitors true 2024-04-25T06:54:18.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:54:19.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:54:19.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:54:19.315 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:54:19.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:54:19.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:54:19.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:54:19.780 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/598905682,v1:127.0.0.1:6803/598905682] [v2:127.0.0.1:6804/598905682,v1:127.0.0.1:6805/598905682] exists,up 5ca31b24-37e3-4416-a84f-76733a34590b 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:54:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:54:19.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T06:54:19.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T06:54:19.787 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T06:54:19.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:54:19.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T06:54:19.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T06:54:19.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T06:54:19.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T06:54:19.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T06:54:19.825 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:19.823+0000 7f6eb28397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:54:19.861 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:19.859+0000 7f6eb28397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:54:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:19.859+0000 7f6eb28397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:54:20.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T06:54:20.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:54:20.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T06:54:20.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:54:20.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:54:20.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:54:20.203 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:54:20.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:54:20.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:54:20.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:54:20.419 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:20.415+0000 7f6eb28397c0 -1 Falling back to public interface 2024-04-25T06:54:20.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:54:21.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:54:21.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:54:21.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:54:21.647 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:54:21.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:54:21.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:54:22.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:54:23.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:54:23.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:54:23.066 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:54:23.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:54:23.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:54:23.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:54:23.288 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:54:23.282+0000 7f6eb28397c0 -1 osd.2 29 log_to_monitors true 2024-04-25T06:54:23.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:54:24.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:54:24.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:54:24.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:54:24.501 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:54:24.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:54:24.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:54:24.911 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/1559876922,v1:127.0.0.1:6811/1559876922] [v2:127.0.0.1:6812/1559876922,v1:127.0.0.1:6813/1559876922] exists,up 9082884f-9bb9-4ce7-8e79-15c15c6660a6 2024-04-25T06:54:24.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:54:24.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:54:24.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:54:24.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-04-25T06:54:24.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-04-25T06:54:25.494 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already in. 2024-04-25T06:54:25.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-04-25T06:54:25.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-04-25T06:54:26.054 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already in. 2024-04-25T06:54:26.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T06:54:26.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-04-25T06:54:26.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T06:54:26.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T06:54:26.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T06:54:26.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T06:54:26.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T06:54:26.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T06:54:26.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T06:54:26.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T06:54:26.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T06:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T06:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T06:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T06:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T06:54:26.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:54:26.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:54:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:54:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:54:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:54:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:54:26.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:54:26.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:54:26.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822660 2024-04-25T06:54:26.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822660 2024-04-25T06:54:26.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660' 2024-04-25T06:54:26.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:54:26.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:54:26.867 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T06:54:26.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T06:54:26.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T06:54:26.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T06:54:26.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:54:26.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:54:27.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757251 2024-04-25T06:54:27.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757251 2024-04-25T06:54:27.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-04-25T06:54:27.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:54:27.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822660 2024-04-25T06:54:27.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:54:27.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:54:27.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822660 2024-04-25T06:54:27.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:54:27.025 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 154618822660 2024-04-25T06:54:27.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822660 2024-04-25T06:54:27.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-04-25T06:54:27.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:54:27.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822658 -lt 154618822660 2024-04-25T06:54:27.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T06:54:28.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T06:54:28.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:54:28.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822660 -lt 154618822660 2024-04-25T06:54:28.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:54:28.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757251 2024-04-25T06:54:28.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:54:28.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:54:28.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757251 2024-04-25T06:54:28.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:54:28.908 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 163208757251 2024-04-25T06:54:28.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757251 2024-04-25T06:54:28.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-04-25T06:54:28.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:54:29.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757251 -lt 163208757251 2024-04-25T06:54:29.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T06:54:29.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:29.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:29.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T06:54:29.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:54:29.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:54:29.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:54:29.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:54:29.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:54:29.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:54:29.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:54:30.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:54:30.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:54:30.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:30.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:54:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T06:54:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T06:54:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T06:54:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:54:30.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T06:54:30.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:54:30.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:54:30.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:54:30.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:54:30.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:54:30.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:54:30.987 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:54:30.987 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:54:31.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:54:31.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:54:31.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:31.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:31.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:54:31.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:54:31.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:54:31.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:54:31.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:54:31.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:54:31.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:54:31.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:54:31.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:54:32.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:54:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:54:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T06:54:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:54:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T06:54:32.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:54:32.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:54:32.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:54:32.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:54:32.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:54:32.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:54:32.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:54:32.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:54:33.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:54:33.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:54:33.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:33.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:54:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:54:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:54:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:54:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:54:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:54:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:54:33.678 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:54:33.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:54:34.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:54:34.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:54:34.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T06:54:34.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:54:34.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T06:54:34.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:54:34.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:54:34.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:54:34.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:54:34.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:54:34.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:54:34.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:54:34.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:54:35.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:54:35.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:54:35.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:35.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:54:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:54:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:54:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:54:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:54:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:54:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:54:35.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:54:35.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:54:36.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:54:36.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:54:36.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T06:54:36.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:54:36.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T06:54:36.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:54:36.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:54:36.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:54:36.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:54:36.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:54:36.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:54:36.933 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:54:36.933 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:54:37.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:54:37.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:54:37.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:37.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:37.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:54:37.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:54:37.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:54:37.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:54:37.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:54:37.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:54:37.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:54:37.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:54:37.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:54:38.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:54:38.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:54:38.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T06:54:38.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:54:38.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T06:54:40.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:54:40.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:54:40.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:54:40.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:54:40.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:54:40.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:54:40.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:54:40.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:54:40.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:54:40.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:54:40.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:40.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:40.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:54:40.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:54:40.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:54:40.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:54:40.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:54:40.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:54:40.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:54:40.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:54:40.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:54:41.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:54:41.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:54:41.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T06:54:41.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:54:41.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T06:54:44.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:54:44.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:54:44.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:54:44.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:54:44.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:54:44.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:54:44.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:54:44.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:54:45.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:54:45.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:54:45.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:45.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:45.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:54:45.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:54:45.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:54:45.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:54:45.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:54:45.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:54:45.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:54:45.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:54:45.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:54:46.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:54:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:54:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T06:54:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:54:46.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T06:54:52.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:54:52.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:54:52.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:54:52.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:54:52.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:54:52.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:54:52.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:54:52.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:54:53.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:54:53.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:54:53.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:54:53.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:54:53.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:54:53.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:54:53.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:54:53.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:54:53.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:54:53.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:54:53.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:54:53.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:54:53.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:54:54.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:54:54.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:54:54.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T06:54:54.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:54:54.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T06:55:01.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:55:01.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:55:01.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:55:01.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:55:01.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:55:01.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:55:01.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:55:01.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:55:01.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:55:01.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:55:01.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:55:01.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:55:02.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:55:02.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:55:02.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:55:02.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:55:02.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:55:02.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:55:02.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:55:02.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:55:02.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:55:02.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:55:02.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:55:02.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T06:55:02.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T06:55:03.513 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T06:55:03.513 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "f2247752-f650-462e-bf9c-bd7810bcb6b8", 2024-04-25T06:55:03.513 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T06:55:03.513 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-04-25T06:55:03.512028+0000", 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "OSDMAP_FLAGS": { 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T06:55:03.514 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 17 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: "OSD_DOWN": { 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 osds down", 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T06:55:03.515 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "osd.1 (root=default,host=smithi088) is down" 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-04-25T06:55:03.516 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:55:03.517 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.518 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-04-25T06:55:03.519 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:51:41.485478Z", 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T06:51:41.485478Z", 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-04-25T06:55:03.520 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-04-25T06:55:03.521 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-04-25T06:55:03.522 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.522 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-04-25T06:55:03.522 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.522 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-04-25T06:55:03.522 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.522 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-04-25T06:55:03.522 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115", 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T06:55:03.523 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 41, 2024-04-25T06:55:03.524 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T06:51:42.392117+0000", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:54:26.345061+0000", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-04-25T06:54:23.604149+0000", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-04-25T06:54:25.863417+0000", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5800384, 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "nodown", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "noin", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "noout", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-04-25T06:55:03.525 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-04-25T06:55:03.526 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-04-25T06:55:03.526 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.526 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 9, 2024-04-25T06:55:03.526 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T06:55:03.526 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T06:55:03.526 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T06:55:03.526 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-04-25T06:55:03.526 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 3, 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "test", 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T06:53:26.155949+0000", 2024-04-25T06:55:03.527 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 1, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool", 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T06:55:03.528 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 1, 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 1, 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 1, 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 1, 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 1, 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T06:55:03.529 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "29", 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 0, 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 0, 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T06:55:03.530 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T06:55:03.531 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T06:55:03.532 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {}, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 2.9900000095367432, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 2.9900000095367432, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T06:55:03.533 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 2, 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 2, 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:55:03.534 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "5ca31b24-37e3-4416-a84f-76733a34590b", 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 5, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 29, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 36, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 40, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 30, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:55:03.535 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 598905682 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 598905682 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.536 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 598905682 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-04-25T06:55:03.537 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 598905682 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6808", 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 598905682 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.538 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6809", 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 598905682 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 598905682 2024-04-25T06:55:03.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 598905682 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6803/598905682", 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6805/598905682", 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6809/598905682", 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6807/598905682", 2024-04-25T06:55:03.540 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "58ee56f4-4e3c-499a-8a46-1f4513ef1de8", 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 0, 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T06:55:03.541 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 10, 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 23, 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 31, 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 228050032 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.542 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 228050032 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T06:55:03.543 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 228050032 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 228050032 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.544 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 228050032 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 228050032 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.545 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 228050032 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 228050032 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.546 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/228050032", 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/228050032", 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/228050032", 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/228050032", 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "exists" 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "9082884f-9bb9-4ce7-8e79-15c15c6660a6", 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:55:03.547 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 15, 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 29, 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 38, 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 0, 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 32, 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T06:55:03.548 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1559876922 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1559876922 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.549 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1559876922 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1559876922 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.550 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1559876922 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1559876922 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:55:03.551 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1559876922 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1559876922 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.552 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/1559876922", 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/1559876922", 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1559876922", 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1559876922", 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.553 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:54:11.694692+0000", 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:52:14.728106+0000", 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:55:03.554 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:54:12.265080+0000", 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:52:47.849309+0000", 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:54:12.806033+0000", 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:55:03.555 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:53:21.856796+0000", 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-04-25T06:55:03.556 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [], 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [], 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-04-25T06:55:03.557 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:55:03.558 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6804/598905682,v1:127.0.0.1:6805/598905682]", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:55:03.559 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:55:03.560 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:51:48.835089Z", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6802/598905682,v1:127.0.0.1:6803/598905682]", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/598905682,v1:127.0.0.1:6809/598905682]", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/598905682,v1:127.0.0.1:6807/598905682]", 2024-04-25T06:55:03.561 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:55:03.562 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/228050032,v1:127.0.0.1:6813/228050032]", 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:55:03.563 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:55:03.564 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:52:21.510785Z", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:55:03.565 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/228050032,v1:127.0.0.1:6811/228050032]", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/228050032,v1:127.0.0.1:6817/228050032]", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/228050032,v1:127.0.0.1:6815/228050032]", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:55:03.566 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/1559876922,v1:127.0.0.1:6813/1559876922]", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:55:03.567 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:55:03.568 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:52:54.673960Z", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:55:03.569 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/1559876922,v1:127.0.0.1:6811/1559876922]", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1559876922,v1:127.0.0.1:6817/1559876922]", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1559876922,v1:127.0.0.1:6815/1559876922]", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:55:03.570 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.571 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [] 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 36 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 38 2024-04-25T06:55:03.572 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 41, 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.573 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.1", 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.2", 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.574 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.575 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.576 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.577 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.578 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:55:03.579 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:55:03.580 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T06:55:03.581 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:55:03.582 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-04-25T06:55:03.583 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:55:03.584 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.585 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-04-25T06:55:03.586 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-04-25T06:55:03.586 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T06:55:03.586 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T06:55:03.586 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.586 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T06:55:03.586 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T06:55:03.586 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T06:55:03.589 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.589 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.589 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-04-25T06:55:03.590 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-04-25T06:55:03.591 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-04-25T06:55:03.592 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-04-25T06:55:03.592 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 0, 2024-04-25T06:55:03.592 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-04-25T06:55:03.592 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-04-25T06:55:03.592 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-04-25T06:55:03.592 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-04-25T06:55:03.592 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-04-25T06:55:03.592 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-04-25T06:55:03.593 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-04-25T06:55:03.594 INFO:tasks.workunit.client.0.smithi088.stderr:report 537038719 2024-04-25T06:55:03.594 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-04-25T06:55:03.594 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-04-25T06:55:03.594 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-04-25T06:55:03.595 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 5, 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 3 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T06:55:03.596 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 1, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 3, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 61700, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 100, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 300, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 100, 2024-04-25T06:55:03.597 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 100, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 100, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 100, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T06:55:03.598 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T06:55:03.599 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T06:55:03.600 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 16, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 16, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 2, 2024-04-25T06:55:03.601 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 2, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 83624, 2024-04-25T06:55:03.602 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2112, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81461, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314489176, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036916224, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2162688, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 460806, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:55:03.603 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10398, 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83416930 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 1, 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.604 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T06:55:03.605 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 83624, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2112, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81461, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314489176, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T06:55:03.606 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036916224, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2162688, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 460806, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10398, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83416930 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T06:55:03.607 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 1, 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T06:55:03.608 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 61700, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 100, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 300, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T06:55:03.609 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 100, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 100, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 100, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 100, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T06:55:03.610 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T06:55:03.611 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T06:55:03.612 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1228800, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 185100, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 16, 2024-04-25T06:55:03.613 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 16, 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 2, 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 2, 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 3 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 154618822667 2024-04-25T06:55:03.614 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 42949672987 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 163208757258 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.615 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 1 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:55:03.616 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 1, 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 0, 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-04-25T06:55:03.617 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 5, 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:54:15.813910+0000", 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-04-25T06:55:03.618 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 89, 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 381, 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 518, 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-04-25T06:55:03.619 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:55:03.620 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T06:55:03.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T06:55:03.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-04-25T06:55:03.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-04-25T06:55:03.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-04-25T06:55:03.621 INFO:tasks.workunit.client.0.smithi088.stdout:writing non-divergent object existing_4 2024-04-25T06:55:03.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:03.907 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:03.907 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T06:54:26.629995+0000 29'100 41:128 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:07:01.038033+0000 0 0 2024-04-25T06:55:03.907 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:03.907 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:03.907 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:03.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T06:55:04.017 INFO:tasks.workunit.client.0.smithi088.stdout:delay recovery 2024-04-25T06:55:04.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-04-25T06:55:04.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:04.396 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:04.396 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T06:54:26.629995+0000 29'100 41:128 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:07:01.038033+0000 0 0 2024-04-25T06:55:04.396 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:04.396 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:04.397 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:04.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-04-25T06:55:04.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-04-25T06:55:04.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T06:55:04.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T06:55:04.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:55:04.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:04.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:04.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.0.asok 2024-04-25T06:55:04.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-04-25T06:55:04.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T06:55:04.513 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T06:55:04.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-04-25T06:55:04.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-04-25T06:55:04.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T06:55:04.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T06:55:04.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:55:04.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:04.525 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:04.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.2.asok 2024-04-25T06:55:04.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-04-25T06:55:04.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T06:55:04.637 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T06:55:04.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-04-25T06:55:04.649 INFO:tasks.workunit.client.0.smithi088.stdout:revive divergent 1 2024-04-25T06:55:04.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:05.031 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:05.031 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T06:54:26.629995+0000 29'100 41:128 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:07:01.038033+0000 0 0 2024-04-25T06:55:05.031 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:05.031 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:05.032 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:05.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-04-25T06:55:05.733 INFO:tasks.workunit.client.0.smithi088.stderr:noup is set 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T06:55:05.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:05.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:55:05.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T06:55:05.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T06:55:05.758 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T06:55:05.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:55:05.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T06:55:05.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T06:55:05.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T06:55:05.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T06:55:05.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T06:55:05.807 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:05.802+0000 7fd322d9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:05.810 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:05.806+0000 7fd322d9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:05.811 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:05.806+0000 7fd322d9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:06.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T06:55:06.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-04-25T06:55:06.355 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:06.350+0000 7fd322d9b7c0 -1 Falling back to public interface 2024-04-25T06:55:09.251 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:09.246+0000 7fd322d9b7c0 -1 osd.1 29 log_to_monitors true 2024-04-25T06:55:11.224 INFO:tasks.workunit.client.0.smithi088.stdout:delay recovery divergent 2024-04-25T06:55:11.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-04-25T06:55:11.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:11.623 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:11.623 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T06:54:26.629995+0000 41'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:07:01.038033+0000 0 0 2024-04-25T06:55:11.623 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:11.623 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:11.623 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:11.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-04-25T06:55:11.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T06:55:11.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T06:55:11.639 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:55:11.639 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:11.639 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:11.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.1.asok 2024-04-25T06:55:11.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-04-25T06:55:11.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T06:55:11.751 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T06:55:11.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-04-25T06:55:12.419 INFO:tasks.workunit.client.0.smithi088.stderr:noup is unset 2024-04-25T06:55:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-04-25T06:55:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:55:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T06:55:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:55:12.440 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:55:12.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:55:12.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:12.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:55:12.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:12.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:55:13.048 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/598905682,v1:127.0.0.1:6803/598905682] [v2:127.0.0.1:6804/598905682,v1:127.0.0.1:6805/598905682] exists,up 5ca31b24-37e3-4416-a84f-76733a34590b 2024-04-25T06:55:13.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:55:13.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:55:13.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:55:13.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-04-25T06:55:13.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:55:13.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T06:55:13.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:55:13.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:55:13.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:13.052 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:55:13.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:55:13.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:13.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:55:13.484 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1961130382,v1:127.0.0.1:6819/1961130382] [v2:127.0.0.1:6820/1961130382,v1:127.0.0.1:6821/1961130382] exists,up 58ee56f4-4e3c-499a-8a46-1f4513ef1de8 2024-04-25T06:55:13.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:13.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:55:13.487 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:55:13.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:13.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:55:13.979 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/1559876922,v1:127.0.0.1:6811/1559876922] [v2:127.0.0.1:6812/1559876922,v1:127.0.0.1:6813/1559876922] exists,up 9082884f-9bb9-4ce7-8e79-15c15c6660a6 2024-04-25T06:55:13.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:55:13.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:55:13.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:55:13.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:14.386 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:14.387 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T06:54:26.629995+0000 41'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:07:01.038033+0000 0 0 2024-04-25T06:55:14.387 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:14.387 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:14.387 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:14.404 INFO:tasks.workunit.client.0.smithi088.stdout:wait for peering 2024-04-25T06:55:14.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-04-25T06:55:14.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:14.790 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:14.790 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T06:54:26.629995+0000 41'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:07:01.038033+0000 0 0 2024-04-25T06:55:14.790 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:14.790 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:14.791 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:14.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-04-25T06:55:14.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-04-25T06:55:14.941 INFO:tasks.workunit.client.0.smithi088.stdout:killing divergent 1 2024-04-25T06:55:14.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:15.343 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:15.343 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T06:54:26.629995+0000 41'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:07:01.038033+0000 0 0 2024-04-25T06:55:15.344 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:15.344 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:15.344 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:15.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T06:55:15.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:55:15.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:55:15.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:55:15.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:55:15.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:55:15.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:55:15.467 INFO:tasks.workunit.client.0.smithi088.stdout:reviving divergent 1 2024-04-25T06:55:15.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-04-25T06:55:15.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:15.842 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:15.842 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 2 2 0 0 62050 0 0 17 84 17 active+recovering+degraded 2024-04-25T06:55:13.380879+0000 41'101 47:141 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 no scrub is scheduled 0 0 2024-04-25T06:55:15.843 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:15.843 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:15.843 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:15.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-04-25T06:55:15.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T06:55:15.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:55:15.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T06:55:15.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:55:15.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:55:15.861 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:55:15.861 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:15.861 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T06:55:15.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T06:55:15.863 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T06:55:15.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:55:15.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T06:55:15.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T06:55:15.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T06:55:15.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T06:55:15.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T06:55:15.895 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:15.894+0000 7f1742ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:15.897 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:15.898+0000 7f1742ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:15.899 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:15.898+0000 7f1742ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T06:55:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:55:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T06:55:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:55:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:55:16.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:16.388 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:55:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:55:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:55:16.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:16.467 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:16.466+0000 7f1742ec57c0 -1 Falling back to public interface 2024-04-25T06:55:16.844 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1961130382,v1:127.0.0.1:6819/1961130382] [v2:127.0.0.1:6820/1961130382,v1:127.0.0.1:6821/1961130382] exists,up 58ee56f4-4e3c-499a-8a46-1f4513ef1de8 2024-04-25T06:55:16.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:55:16.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:55:16.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:55:16.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-04-25T06:55:19.140 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:19.138+0000 7f1742ec57c0 -1 osd.1 47 log_to_monitors true 2024-04-25T06:55:36.847 INFO:tasks.workunit.client.0.smithi088.stdout:allowing recovery 2024-04-25T06:55:36.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-04-25T06:55:36.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:37.262 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:37.262 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-04-25T06:55:20.739217+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:38:38.389861+0000 0 0 2024-04-25T06:55:37.262 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:37.262 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:37.263 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:37.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T06:55:37.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T06:55:37.434 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T06:55:37.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T06:55:37.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T06:55:37.592 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T06:55:37.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T06:55:37.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T06:55:37.742 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T06:55:37.755 INFO:tasks.workunit.client.0.smithi088.stdout:reading divergent objects 2024-04-25T06:55:37.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-04-25T06:55:37.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-04-25T06:55:38.137 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:55:38.138 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-04-25T06:55:20.739217+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:53:26.188905+0000 0'0 2024-04-25T06:53:26.188905+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:38:38.389861+0000 0 0 2024-04-25T06:55:38.138 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:55:38.138 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:55:38.138 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:55:38.153 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-04-25T06:55:38.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-04-25T06:55:38.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T06:55:38.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T06:55:38.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T06:55:38.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T06:55:38.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T06:55:38.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T06:55:38.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T06:55:38.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T06:55:38.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-04-25T06:55:38.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-04-25T06:55:38.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T06:55:38.304 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4614.0:1 2024-04-25T06:54:00.136356+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4614.0:1 2024-04-25T06:54:00.136356+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4614.0:1 2024-04-25T06:54:00.136356+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:55:12.926+0000 7fd306d25640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-04-25T06:55:38.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-04-25T06:55:38.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T06:55:38.310 INFO:tasks.workunit.client.0.smithi088.stdout:success 2024-04-25T06:55:38.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-04-25T06:55:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-04-25T06:55:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T06:55:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T06:55:38.839 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T06:55:38.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-04-25T06:55:38.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:55:38.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:55:38.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:55:38.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:55:38.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:55:44.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:55:44.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T06:55:44.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T06:55:44.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T06:55:44.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T06:55:44.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:55:44.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:55:44.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:55:44.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:55:44.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:55:44.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:55:44.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T06:55:44.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T06:55:44.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T06:55:44.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T06:55:44.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T06:55:44.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T06:55:44.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:55:44.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T06:55:44.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T06:55:44.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:55:44.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:55:44.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T06:55:44.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T06:55:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T06:55:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T06:55:44.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T06:55:44.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T06:55:44.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T06:55:44.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T06:55:44.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:55:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:55:44.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:55:44.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T06:55:44.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T06:55:44.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T06:55:44.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T06:55:44.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T06:55:44.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T06:55:44.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:55:44.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T06:55:44.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T06:55:44.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:55:44.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:55:44.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T06:55:44.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T06:55:44.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T06:55:44.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:44.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:44.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T06:55:44.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T06:55:44.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T06:55:44.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T06:55:44.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T06:55:44.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:44.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:44.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26017 2024-04-25T06:55:44.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T06:55:44.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T06:55:44.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T06:55:44.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T06:55:44.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-04-25T06:55:44.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-04-25T06:55:44.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-04-25T06:55:44.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-04-25T06:55:44.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-04-25T06:55:44.227 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-04-25T06:55:44.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-04-25T06:55:44.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-04-25T06:55:44.228 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:55:44.228 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:55:44.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-04-25T06:55:44.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T06:55:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T06:55:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T06:55:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T06:55:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T06:55:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T06:55:44.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T06:55:44.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:55:44.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:55:44.707 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:55:44.707 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:44.707 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:44.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:55:44.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T06:55:44.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T06:55:44.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T06:55:44.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T06:55:44.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T06:55:44.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T06:55:44.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T06:55:44.904 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T06:55:44.904 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T06:55:44.904 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T06:55:44.904 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:55:44.904 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:44.904 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:44.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-mon.a.asok 2024-04-25T06:55:44.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T06:55:44.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26017/ceph-mon.a.asok config get fsid 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T06:55:45.021 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:55:45.022 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:45.022 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:45.022 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-mon.a.asok 2024-04-25T06:55:45.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T06:55:45.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26017/ceph-mon.a.asok config get mon_host 2024-04-25T06:55:45.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-04-25T06:55:45.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T06:55:45.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T06:55:45.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T06:55:45.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T06:55:45.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T06:55:45.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T06:55:46.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T06:55:46.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:55:46.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:55:46.048 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:55:46.048 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:46.048 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:46.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:55:46.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T06:55:46.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:55:46.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:55:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T06:55:46.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:55:46.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6 2024-04-25T06:55:46.094 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6 2024-04-25T06:55:46.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6' 2024-04-25T06:55:46.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:55:46.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBy/ilm/XbuBhAA1Va8dhLO54o5wqXJqyYVYQ== 2024-04-25T06:55:46.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBy/ilm/XbuBhAA1Va8dhLO54o5wqXJqyYVYQ=="}' 2024-04-25T06:55:46.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6 -i td/divergent-priors/0/new.json 2024-04-25T06:55:46.412 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:55:46.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T06:55:46.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBy/ilm/XbuBhAA1Va8dhLO54o5wqXJqyYVYQ== --osd-uuid ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6 2024-04-25T06:55:46.475 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:46.473+0000 7ff056e437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:46.480 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:46.477+0000 7ff056e437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:46.481 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:46.481+0000 7ff056e437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:46.481 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:46.481+0000 7ff056e437c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T06:55:46.482 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:46.481+0000 7ff056e437c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T06:55:46.482 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:46.481+0000 7ff056e437c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T06:55:46.520 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:46.521+0000 7ff056e437c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T06:55:52.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T06:55:52.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:55:52.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T06:55:52.409 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T06:55:52.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:55:52.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T06:55:52.993 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T06:55:52.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:55:52.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:55:52.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:55:52.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:55:53.034 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:53.033+0000 7f1fb7a6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:53.046 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:53.045+0000 7f1fb7a6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:53.047 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:53.045+0000 7f1fb7a6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:55:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T06:55:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:55:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T06:55:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:55:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:55:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:53.440 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:55:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:55:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:55:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:53.625+0000 7f1fb7a6e7c0 -1 Falling back to public interface 2024-04-25T06:55:53.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:55:54.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:55:54.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:54.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:55:54.868 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:55:54.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:54.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:55:55.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:55:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:55:56.149+0000 7f1fb7a6e7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T06:55:56.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:55:56.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:56.317 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:55:56.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:55:56.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:56.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:55:56.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:55:57.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:55:57.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:57.894 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:55:57.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:55:57.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:57.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:55:58.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:55:59.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:55:59.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:55:59.354 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:55:59.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:55:59.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:55:59.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:55:59.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:00.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:00.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:00.854 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:56:00.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:56:00.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:00.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:01.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:02.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:02.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:02.272 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:56:02.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:56:02.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:02.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:02.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:03.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:03.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:03.705 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:56:03.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:56:03.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:03.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:04.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:05.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:05.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:05.207 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:56:05.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:56:05.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:05.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:05.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:06.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:06.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:06.662 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:56:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:56:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:07.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:08.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:08.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:08.197 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:56:08.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:56:08.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:08.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:09.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:10.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:10.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:10.014 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:56:10.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:56:10.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:10.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:10.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:11.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:11.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:11.839 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:56:11.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:56:11.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:11.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:12.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:13.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:13.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:13.427 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:56:13.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:56:13.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:13.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:14.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:15.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:15.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:15.246 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:56:15.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:56:15.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:15.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:15.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:16.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:16.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:16.786 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:56:16.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:56:16.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:16.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:17.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:18.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:18.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:18.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T06:56:18.433 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T06:56:18.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:18.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:56:18.896 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/892493043,v1:127.0.0.1:6803/892493043] [v2:127.0.0.1:6804/892493043,v1:127.0.0.1:6805/892493043] exists,up ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6 2024-04-25T06:56:18.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:56:18.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:56:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:56:18.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:56:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T06:56:18.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:56:18.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2357bb75-0d21-45fe-8b0a-60ccd011d618 2024-04-25T06:56:18.903 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 2357bb75-0d21-45fe-8b0a-60ccd011d618 2024-04-25T06:56:18.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2357bb75-0d21-45fe-8b0a-60ccd011d618' 2024-04-25T06:56:18.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:56:18.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCS/ilmLmdzNxAAUiAQl4R21GT4umWtxKjePw== 2024-04-25T06:56:18.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCS/ilmLmdzNxAAUiAQl4R21GT4umWtxKjePw=="}' 2024-04-25T06:56:18.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2357bb75-0d21-45fe-8b0a-60ccd011d618 -i td/divergent-priors/1/new.json 2024-04-25T06:56:19.395 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:56:19.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T06:56:19.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCS/ilmLmdzNxAAUiAQl4R21GT4umWtxKjePw== --osd-uuid 2357bb75-0d21-45fe-8b0a-60ccd011d618 2024-04-25T06:56:19.455 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:19.453+0000 7f71c79e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:19.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:19.457+0000 7f71c79e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:19.458 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:19.457+0000 7f71c79e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:19.459 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:19.457+0000 7f71c79e37c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T06:56:19.459 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:19.457+0000 7f71c79e37c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T06:56:19.459 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:19.457+0000 7f71c79e37c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T06:56:19.488 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:19.489+0000 7f71c79e37c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T06:56:25.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T06:56:25.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:56:25.257 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T06:56:25.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T06:56:25.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:56:25.823 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T06:56:25.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T06:56:25.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:56:25.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:56:25.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:56:25.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:56:25.856 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:25.857+0000 7fcfd1a1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:25.869 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:25.869+0000 7fcfd1a1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:25.870 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:25.869+0000 7fcfd1a1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:26.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T06:56:26.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:56:26.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T06:56:26.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:56:26.240 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:56:26.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:56:26.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:26.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:56:26.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:26.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:26.426 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:26.425+0000 7fcfd1a1a7c0 -1 Falling back to public interface 2024-04-25T06:56:26.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:27.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:27.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:27.637 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:56:27.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:56:27.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:27.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:28.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:28.959 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:28.957+0000 7fcfd1a1a7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T06:56:29.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:29.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:29.061 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:56:29.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:56:29.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:29.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:29.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:30.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:30.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:30.643 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:56:30.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:56:30.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:30.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:31.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:32.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:32.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:32.120 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:56:32.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:56:32.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:32.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:32.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:33.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:33.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:33.593 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:56:33.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:56:33.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:33.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:34.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:35.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:35.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:35.144 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:56:35.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:56:35.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:35.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:35.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:36.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:36.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:36.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:56:36.749 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:56:36.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:36.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:37.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:38.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:38.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:38.226 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:56:38.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:56:38.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:38.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:38.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:39.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:39.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:39.756 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:56:39.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:56:39.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:39.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:40.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:41.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:41.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:41.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:56:41.176 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:56:41.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:41.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:42.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:43.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:43.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:43.149 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:56:43.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:56:43.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:43.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:43.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:44.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:44.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:44.839 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:56:44.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:56:44.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:44.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:45.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:46.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:46.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:46.845 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:56:46.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:56:46.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:46.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:47.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:48.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:48.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:48.747 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:56:48.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:56:48.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:48.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:50.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:51.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:51.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:51.136 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:56:51.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:56:51.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:51.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:51.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:56:52.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:56:52.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:56:52.901 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T06:56:52.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T06:56:52.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:56:52.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:56:53.341 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4024521766,v1:127.0.0.1:6811/4024521766] [v2:127.0.0.1:6812/4024521766,v1:127.0.0.1:6813/4024521766] exists,up 2357bb75-0d21-45fe-8b0a-60ccd011d618 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T06:56:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:56:53.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:56:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:56:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T06:56:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T06:56:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T06:56:53.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T06:56:53.396 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 e77a9999-9ad8-4555-a6ca-57b80790a230 2024-04-25T06:56:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e77a9999-9ad8-4555-a6ca-57b80790a230 2024-04-25T06:56:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e77a9999-9ad8-4555-a6ca-57b80790a230' 2024-04-25T06:56:53.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T06:56:53.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1/ilmns2AGRAAIw3RZqg6KqxgSPxEaZ2BwQ== 2024-04-25T06:56:53.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1/ilmns2AGRAAIw3RZqg6KqxgSPxEaZ2BwQ=="}' 2024-04-25T06:56:53.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e77a9999-9ad8-4555-a6ca-57b80790a230 -i td/divergent-priors/2/new.json 2024-04-25T06:56:53.947 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:56:53.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T06:56:53.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1/ilmns2AGRAAIw3RZqg6KqxgSPxEaZ2BwQ== --osd-uuid e77a9999-9ad8-4555-a6ca-57b80790a230 2024-04-25T06:56:54.004 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:54.004+0000 7f166749c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:54.019 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:54.016+0000 7f166749c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:54.020 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:54.020+0000 7f166749c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:56:54.021 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:54.020+0000 7f166749c7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T06:56:54.021 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:54.020+0000 7f166749c7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T06:56:54.021 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:54.020+0000 7f166749c7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T06:56:54.094 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:56:54.092+0000 7f166749c7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T06:57:00.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T06:57:00.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T06:57:00.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T06:57:00.017 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T06:57:00.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T06:57:00.598 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T06:57:00.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T06:57:00.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:57:00.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T06:57:00.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T06:57:00.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T06:57:00.640 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:57:00.640+0000 7fe521ae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:57:00.660 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:57:00.660+0000 7fe521ae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:57:00.661 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:57:00.660+0000 7fe521ae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:57:01.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T06:57:01.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:57:01.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T06:57:01.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:57:01.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:57:01.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:01.092 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:57:01.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:57:01.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:01.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:01.229 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:57:01.228+0000 7fe521ae17c0 -1 Falling back to public interface 2024-04-25T06:57:01.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:02.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:02.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:02.526 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:57:02.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:57:02.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:02.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:03.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:03.784 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:57:03.784+0000 7fe521ae17c0 -1 osd.2 0 log_to_monitors true 2024-04-25T06:57:04.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:04.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:04.014 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:57:04.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:57:04.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:04.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:04.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:05.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:05.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:05.682 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:57:05.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:57:05.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:05.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:06.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:07.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:07.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:07.123 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T06:57:07.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T06:57:07.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:07.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:07.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:08.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:08.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:08.556 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T06:57:08.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T06:57:08.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:08.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:09.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:10.025 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T06:57:10.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T06:57:10.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:10.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:10.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:11.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:11.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:11.568 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T06:57:11.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T06:57:11.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:11.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:12.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:13.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:13.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:13.028 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T06:57:13.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T06:57:13.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:13.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:13.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:14.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:14.543 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T06:57:14.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T06:57:14.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:14.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:15.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:16.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:16.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:16.049 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T06:57:16.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T06:57:16.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:16.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:16.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:17.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:17.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:17.583 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T06:57:17.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T06:57:17.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:17.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:18.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:19.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:19.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:19.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T06:57:19.553 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T06:57:19.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:19.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:20.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:21.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:21.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:21.410 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T06:57:21.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T06:57:21.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:21.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:22.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:23.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:23.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:23.463 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T06:57:23.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T06:57:23.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:23.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:25.518 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T06:57:25.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T06:57:25.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:25.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:26.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:57:27.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:57:27.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:57:27.396 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T06:57:27.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T06:57:27.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:57:27.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:57:27.865 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2170863418,v1:127.0.0.1:6819/2170863418] [v2:127.0.0.1:6820/2170863418,v1:127.0.0.1:6821/2170863418] exists,up e77a9999-9ad8-4555-a6ca-57b80790a230 2024-04-25T06:57:27.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:57:27.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:57:27.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:57:27.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-04-25T06:57:28.538 INFO:tasks.workunit.client.0.smithi088.stderr:noout is set 2024-04-25T06:57:28.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-04-25T06:57:29.425 INFO:tasks.workunit.client.0.smithi088.stderr:noin is set 2024-04-25T06:57:29.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-04-25T06:57:30.186 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T06:57:30.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-04-25T06:57:30.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T06:57:30.744 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T06:57:30.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T06:57:31.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-04-25T06:57:32.730 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 3 2024-04-25T06:57:32.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-04-25T06:57:33.633 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 2 2024-04-25T06:57:33.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-04-25T06:57:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:57:33.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:57:34.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:57:34.179 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:57:34.180 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:57:34.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:57:34.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:34.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:57:34.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-04-25T06:57:34.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-04-25T06:57:34.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-04-25T06:57:34.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:34.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:57:34.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705680 2024-04-25T06:57:34.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705680 2024-04-25T06:57:34.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-38654705680' 2024-04-25T06:57:34.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:34.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:57:34.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542153 2024-04-25T06:57:34.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542153 2024-04-25T06:57:34.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-38654705680 2-60129542153' 2024-04-25T06:57:34.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:57:34.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-04-25T06:57:34.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:57:34.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:57:34.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-04-25T06:57:34.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:57:34.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-04-25T06:57:34.675 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836500 2024-04-25T06:57:34.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-25T06:57:34.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:57:35.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836500 2024-04-25T06:57:35.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:57:35.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705680 2024-04-25T06:57:35.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:57:35.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T06:57:35.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705680 2024-04-25T06:57:35.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:57:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705680 2024-04-25T06:57:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2024-04-25T06:57:35.126 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 38654705680 2024-04-25T06:57:35.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T06:57:35.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705680 -lt 38654705680 2024-04-25T06:57:35.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:57:35.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542153 2024-04-25T06:57:35.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:57:35.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:57:35.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542153 2024-04-25T06:57:35.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:57:35.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542153 2024-04-25T06:57:35.597 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 60129542153 2024-04-25T06:57:35.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2024-04-25T06:57:35.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:57:36.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542152 -lt 60129542153 2024-04-25T06:57:36.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T06:57:37.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T06:57:37.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:57:37.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542153 -lt 60129542153 2024-04-25T06:57:37.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-04-25T06:57:37.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T06:57:37.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T06:57:37.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T06:57:37.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T06:57:37.487 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T06:57:37.487 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T06:57:37.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T06:57:37.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T06:57:37.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T06:57:37.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T06:57:37.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T06:57:37.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T06:57:37.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T06:57:37.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:57:37.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:57:38.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:57:38.002 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:57:38.002 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:57:38.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:57:38.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:38.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:57:38.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-04-25T06:57:38.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-04-25T06:57:38.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-04-25T06:57:38.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:38.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:57:38.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705683 2024-04-25T06:57:38.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705683 2024-04-25T06:57:38.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-38654705683' 2024-04-25T06:57:38.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:38.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:57:38.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542156 2024-04-25T06:57:38.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542156 2024-04-25T06:57:38.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-38654705683 2-60129542156' 2024-04-25T06:57:38.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:57:38.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-04-25T06:57:38.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:57:38.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:57:38.476 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-04-25T06:57:38.476 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:57:38.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-04-25T06:57:38.477 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836503 2024-04-25T06:57:38.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-25T06:57:38.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:57:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836503 2024-04-25T06:57:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:57:38.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705683 2024-04-25T06:57:38.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:57:38.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T06:57:38.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705683 2024-04-25T06:57:38.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:57:38.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705683 2024-04-25T06:57:38.906 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 38654705683 2024-04-25T06:57:38.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2024-04-25T06:57:38.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T06:57:39.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705683 -lt 38654705683 2024-04-25T06:57:39.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:57:39.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542156 2024-04-25T06:57:39.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:57:39.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:57:39.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542156 2024-04-25T06:57:39.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:57:39.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542156 2024-04-25T06:57:39.378 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 60129542156 2024-04-25T06:57:39.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2024-04-25T06:57:39.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:57:39.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542156 -lt 60129542156 2024-04-25T06:57:39.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T06:57:39.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:57:39.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:57:40.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T06:57:40.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:57:40.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:57:40.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:57:40.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:57:40.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:57:40.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:57:40.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:57:40.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T06:57:40.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:57:40.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:57:40.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:57:41.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T06:57:41.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T06:57:41.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T06:57:41.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-04-25T06:57:41.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-04-25T06:57:41.676 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:57:41.692 INFO:tasks.workunit.client.0.smithi088.stdout:primary and soon to be divergent is 1 2024-04-25T06:57:41.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-04-25T06:57:41.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-04-25T06:57:41.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:57:42.071 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:57:42.072 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T06:57:32.172349+0000 0'0 28:21 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:35:14.043332+0000 0 0 2024-04-25T06:57:42.072 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:57:42.072 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:57:42.072 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:57:42.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-04-25T06:57:42.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T06:57:42.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-04-25T06:57:42.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-04-25T06:57:42.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T06:57:42.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-04-25T06:57:42.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-04-25T06:57:42.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T06:57:42.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-04-25T06:57:42.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-04-25T06:57:42.089 INFO:tasks.workunit.client.0.smithi088.stdout:writing initial objects 2024-04-25T06:57:42.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-04-25T06:57:42.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-04-25T06:57:42.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:42.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-04-25T06:57:42.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:42.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-04-25T06:57:42.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:42.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-04-25T06:57:42.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:42.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-04-25T06:57:42.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:42.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-04-25T06:57:42.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:42.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-04-25T06:57:43.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:43.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-04-25T06:57:43.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:43.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-04-25T06:57:43.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:43.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-04-25T06:57:43.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:43.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-04-25T06:57:43.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:43.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-04-25T06:57:43.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:43.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-04-25T06:57:43.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:43.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-04-25T06:57:44.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:44.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-04-25T06:57:44.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:44.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-04-25T06:57:44.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:44.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-04-25T06:57:44.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:44.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-04-25T06:57:44.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:44.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-04-25T06:57:44.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:44.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-04-25T06:57:45.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:45.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-04-25T06:57:45.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:45.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-04-25T06:57:45.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:45.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-04-25T06:57:45.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:45.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-04-25T06:57:45.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:45.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-04-25T06:57:45.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:45.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-04-25T06:57:46.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:46.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-04-25T06:57:46.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:46.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-04-25T06:57:46.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:46.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-04-25T06:57:46.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:46.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-04-25T06:57:46.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:46.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-04-25T06:57:46.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:46.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-04-25T06:57:46.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:46.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-04-25T06:57:47.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:47.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-04-25T06:57:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-04-25T06:57:47.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:47.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-04-25T06:57:47.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:47.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-04-25T06:57:47.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:47.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-04-25T06:57:47.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:47.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-04-25T06:57:48.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:48.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-04-25T06:57:48.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:48.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-04-25T06:57:48.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:48.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-04-25T06:57:48.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:48.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-04-25T06:57:48.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:48.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-04-25T06:57:48.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:48.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-04-25T06:57:48.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:48.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-04-25T06:57:49.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:49.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-04-25T06:57:49.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:49.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-04-25T06:57:49.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:49.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-04-25T06:57:49.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:49.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-04-25T06:57:49.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:49.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-04-25T06:57:50.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:50.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-04-25T06:57:50.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:50.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-04-25T06:57:50.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:50.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-04-25T06:57:50.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:50.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-04-25T06:57:50.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:50.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-04-25T06:57:50.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:50.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-04-25T06:57:50.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:50.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-04-25T06:57:51.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:51.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-04-25T06:57:51.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:51.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-04-25T06:57:51.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:51.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-04-25T06:57:51.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:51.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-04-25T06:57:51.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:51.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-04-25T06:57:51.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:51.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-04-25T06:57:52.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:52.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-04-25T06:57:52.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:52.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-04-25T06:57:52.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:52.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-04-25T06:57:52.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:52.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-04-25T06:57:52.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:52.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-04-25T06:57:52.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:52.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-04-25T06:57:53.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:53.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-04-25T06:57:53.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:53.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-04-25T06:57:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-04-25T06:57:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-04-25T06:57:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-04-25T06:57:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-04-25T06:57:53.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:53.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-04-25T06:57:54.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:54.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-04-25T06:57:54.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:54.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-04-25T06:57:54.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:54.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-04-25T06:57:54.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:54.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-04-25T06:57:54.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:54.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-04-25T06:57:54.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:54.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-04-25T06:57:55.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:55.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-04-25T06:57:55.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:55.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-04-25T06:57:55.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:55.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-04-25T06:57:55.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:55.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-04-25T06:57:55.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:55.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-04-25T06:57:55.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:55.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-04-25T06:57:56.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:56.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-04-25T06:57:56.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:56.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-04-25T06:57:56.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:56.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-04-25T06:57:56.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:56.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-04-25T06:57:56.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:56.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-04-25T06:57:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-04-25T06:57:57.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:57.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-04-25T06:57:57.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:57.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-04-25T06:57:57.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:57.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-04-25T06:57:57.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:57.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-04-25T06:57:57.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:57.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-04-25T06:57:57.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T06:57:57.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-04-25T06:57:57.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T06:57:57.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-04-25T06:57:57.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T06:57:57.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T06:57:57.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T06:57:57.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T06:57:57.969 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T06:57:57.969 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T06:57:57.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T06:57:57.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T06:57:57.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T06:57:58.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T06:57:58.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T06:57:58.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T06:57:58.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T06:57:58.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:57:58.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:57:58.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:57:58.529 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:57:58.530 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:57:58.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:57:58.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:58.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:57:58.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836509 2024-04-25T06:57:58.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836509 2024-04-25T06:57:58.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509' 2024-04-25T06:57:58.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:58.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:57:58.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705689 2024-04-25T06:57:58.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705689 2024-04-25T06:57:58.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-38654705689' 2024-04-25T06:57:58.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:57:58.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:57:59.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542162 2024-04-25T06:57:59.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542162 2024-04-25T06:57:59.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-38654705689 2-60129542162' 2024-04-25T06:57:59.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:57:59.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:57:59.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836509 2024-04-25T06:57:59.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:57:59.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-04-25T06:57:59.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:57:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836509 2024-04-25T06:57:59.028 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836509 2024-04-25T06:57:59.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-04-25T06:57:59.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:57:59.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836509 2024-04-25T06:57:59.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T06:58:00.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T06:58:00.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:58:00.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836509 2024-04-25T06:58:00.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:58:00.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705689 2024-04-25T06:58:00.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:58:00.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T06:58:00.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705689 2024-04-25T06:58:00.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:58:00.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705689 2024-04-25T06:58:00.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705689' 2024-04-25T06:58:00.928 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 38654705689 2024-04-25T06:58:00.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T06:58:01.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705689 -lt 38654705689 2024-04-25T06:58:01.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:58:01.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542162 2024-04-25T06:58:01.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:58:01.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:58:01.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542162 2024-04-25T06:58:01.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:58:01.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542162 2024-04-25T06:58:01.392 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 60129542162 2024-04-25T06:58:01.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2024-04-25T06:58:01.393 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:58:01.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542162 -lt 60129542162 2024-04-25T06:58:01.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T06:58:01.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:01.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:02.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T06:58:02.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:02.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:02.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:02.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:02.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:02.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:02.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:02.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T06:58:02.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:02.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:02.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:03.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T06:58:03.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T06:58:03.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T06:58:03.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-04-25T06:58:03.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T06:58:03.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T06:58:03.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T06:58:03.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T06:58:03.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-04-25T06:58:03.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-04-25T06:58:03.886 INFO:tasks.workunit.client.0.smithi088.stdout:blackholing osds 0 2 2024-04-25T06:58:03.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:58:04.306 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:58:04.306 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T06:57:32.172349+0000 28'100 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:35:14.043332+0000 0 0 2024-04-25T06:58:04.306 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:58:04.306 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:58:04.306 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:58:04.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:58:04.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-04-25T06:58:04.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T06:58:04.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T06:58:04.324 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:58:04.324 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:58:04.324 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:58:04.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.0.asok 2024-04-25T06:58:04.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-04-25T06:58:04.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T06:58:04.420 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T06:58:04.420 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T06:58:04.420 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T06:58:04.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:58:04.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-04-25T06:58:04.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T06:58:04.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T06:58:04.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:58:04.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:58:04.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:58:04.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.2.asok 2024-04-25T06:58:04.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-04-25T06:58:04.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T06:58:04.539 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T06:58:04.539 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T06:58:04.539 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T06:58:04.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-04-25T06:58:04.552 INFO:tasks.workunit.client.0.smithi088.stdout:create new divergent objects 2024-04-25T06:58:04.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-04-25T06:58:04.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T06:58:04.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-04-25T06:58:04.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T06:58:04.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-04-25T06:58:04.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-04-25T06:58:04.555 INFO:tasks.workunit.client.0.smithi088.stdout:writing divergent objects 2024-04-25T06:58:04.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-04-25T06:58:04.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T06:58:04.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-04-25T06:58:04.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T06:58:04.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-04-25T06:58:04.557 INFO:tasks.workunit.client.0.smithi088.stdout:remove divergent objects 2024-04-25T06:58:04.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-04-25T06:58:04.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-04-25T06:58:04.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T06:58:04.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-04-25T06:58:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-04-25T06:58:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T06:58:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-04-25T06:58:04.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-04-25T06:58:04.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-04-25T06:58:04.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-04-25T06:58:04.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-04-25T06:58:14.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-04-25T06:58:14.569 INFO:tasks.workunit.client.0.smithi088.stderr:rados: no process found 2024-04-25T06:58:14.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-04-25T06:58:14.570 INFO:tasks.workunit.client.0.smithi088.stdout:killing all the osds 2024-04-25T06:58:14.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:58:14.974 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:58:14.974 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-04-25T06:57:32.172349+0000 28'106 28:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:35:14.043332+0000 0 0 2024-04-25T06:58:14.974 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:58:14.974 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:58:14.974 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:58:14.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-04-25T06:58:14.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:58:14.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:58:14.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:58:14.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:58:14.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:58:15.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:58:15.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T06:58:15.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-04-25T06:58:15.636 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T06:58:15.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T06:58:15.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-04-25T06:58:16.191 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T06:58:16.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T06:58:16.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-04-25T06:58:16.781 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already down. 2024-04-25T06:58:16.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:58:16.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-04-25T06:58:17.334 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T06:58:17.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:58:17.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-04-25T06:58:17.911 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already out. 2024-04-25T06:58:17.927 INFO:tasks.workunit.client.0.smithi088.stdout:bringing up non_divergent 0 2 2024-04-25T06:58:17.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-04-25T06:58:17.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:58:18.327 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:58:18.327 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-04-25T06:57:32.172349+0000 28'106 28:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:35:14.043332+0000 0 0 2024-04-25T06:58:18.327 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:58:18.327 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:58:18.328 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:58:18.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:58:18.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T06:58:18.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T06:58:18.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:58:18.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T06:58:18.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:58:18.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:58:18.346 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:58:18.346 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:58:18.346 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:58:18.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T06:58:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T06:58:18.349 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T06:58:18.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:58:18.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T06:58:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T06:58:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T06:58:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T06:58:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T06:58:18.385 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:18.384+0000 7f6e0608e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:58:18.387 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:18.388+0000 7f6e0608e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:58:18.389 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:18.388+0000 7f6e0608e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:58:18.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T06:58:18.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:58:18.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T06:58:18.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:58:18.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:58:18.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:58:18.786 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:58:18.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:58:18.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:58:18.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:58:18.952 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:18.952+0000 7f6e0608e7c0 -1 Falling back to public interface 2024-04-25T06:58:19.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:58:20.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:58:20.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:58:20.257 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:58:20.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:58:20.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:58:20.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:58:20.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:58:21.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:58:21.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:58:21.781 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:58:21.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:58:21.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:58:21.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:58:21.802 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:21.804+0000 7f6e0608e7c0 -1 osd.0 28 log_to_monitors true 2024-04-25T06:58:22.242 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/4091415756,v1:127.0.0.1:6803/4091415756] [v2:127.0.0.1:6804/4091415756,v1:127.0.0.1:6805/4091415756] exists,up ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6 2024-04-25T06:58:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:58:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:58:22.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:58:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:58:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-04-25T06:58:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T06:58:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T06:58:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T06:58:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T06:58:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T06:58:22.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:58:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:58:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:58:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:58:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T06:58:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T06:58:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T06:58:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T06:58:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:58:22.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:58:22.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:58:22.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:58:22.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:58:22.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:58:22.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:58:22.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:58:22.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T06:58:22.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T06:58:22.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T06:58:22.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T06:58:22.253 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T06:58:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:58:22.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T06:58:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T06:58:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T06:58:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T06:58:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T06:58:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:22.304+0000 7ffb6cc9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:58:22.305 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:22.304+0000 7ffb6cc9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:58:22.307 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:22.308+0000 7ffb6cc9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:58:22.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T06:58:22.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:58:22.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T06:58:22.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:58:22.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:58:22.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:58:22.756 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:58:22.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:58:22.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:58:22.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:58:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:22.888+0000 7ffb6cc9d7c0 -1 Falling back to public interface 2024-04-25T06:58:23.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:58:24.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:58:24.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:58:24.216 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T06:58:24.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T06:58:24.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:58:24.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:58:24.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:58:25.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:58:25.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:58:25.645 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T06:58:25.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T06:58:25.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:58:25.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:58:25.810 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:58:25.812+0000 7ffb6cc9d7c0 -1 osd.2 28 log_to_monitors true 2024-04-25T06:58:26.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T06:58:27.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T06:58:27.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:58:27.069 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T06:58:27.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T06:58:27.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:58:27.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:58:27.490 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [14,28) [v2:127.0.0.1:6810/1330652640,v1:127.0.0.1:6811/1330652640] [v2:127.0.0.1:6812/1330652640,v1:127.0.0.1:6813/1330652640] exists,up e77a9999-9ad8-4555-a6ca-57b80790a230 2024-04-25T06:58:27.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:58:27.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:58:27.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:58:27.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:58:27.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-04-25T06:58:28.079 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already in. 2024-04-25T06:58:28.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:58:28.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-04-25T06:58:28.647 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already in. 2024-04-25T06:58:28.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T06:58:28.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-04-25T06:58:28.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T06:58:28.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T06:58:28.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T06:58:28.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T06:58:28.667 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T06:58:28.667 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T06:58:28.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T06:58:28.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T06:58:28.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T06:58:28.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T06:58:28.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T06:58:28.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T06:58:28.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T06:58:28.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:58:28.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:58:29.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:58:29.187 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:58:29.187 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:58:29.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:58:29.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:58:29.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:58:29.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855364 2024-04-25T06:58:29.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855364 2024-04-25T06:58:29.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855364' 2024-04-25T06:58:29.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:58:29.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:58:29.486 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T06:58:29.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T06:58:29.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T06:58:29.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T06:58:29.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:58:29.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:58:29.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789956 2024-04-25T06:58:29.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789956 2024-04-25T06:58:29.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855364 2-158913789956' 2024-04-25T06:58:29.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:58:29.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855364 2024-04-25T06:58:29.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:58:29.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:58:29.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855364 2024-04-25T06:58:29.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:58:29.641 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 150323855364 2024-04-25T06:58:29.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855364 2024-04-25T06:58:29.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2024-04-25T06:58:29.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:58:30.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855362 -lt 150323855364 2024-04-25T06:58:30.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T06:58:31.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T06:58:31.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:58:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855364 -lt 150323855364 2024-04-25T06:58:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:58:31.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-158913789956 2024-04-25T06:58:31.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:58:31.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:58:31.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-158913789956 2024-04-25T06:58:31.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:58:31.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789956 2024-04-25T06:58:31.542 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 158913789956 2024-04-25T06:58:31.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 158913789956' 2024-04-25T06:58:31.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:58:32.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789956 -lt 158913789956 2024-04-25T06:58:32.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T06:58:32.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:32.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:32.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T06:58:32.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:32.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:32.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:32.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:32.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:32.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:32.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:32.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:58:32.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:32.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:32.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:33.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:58:33.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T06:58:33.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T06:58:33.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T06:58:33.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:58:33.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T06:58:33.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:58:33.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:33.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:33.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:33.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:33.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:33.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:33.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:34.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:58:34.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:34.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:34.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:34.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:58:34.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:58:34.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:58:34.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:58:34.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:58:34.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:58:34.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:58:34.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:58:34.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:58:35.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:58:35.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:58:35.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T06:58:35.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:58:35.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T06:58:35.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:58:35.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:35.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:35.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:35.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:35.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:35.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:35.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:35.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:58:35.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:35.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:35.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:36.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:58:36.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:58:36.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:58:36.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:58:36.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:58:36.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:58:36.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:58:36.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:58:36.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:58:36.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:58:36.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:58:36.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T06:58:36.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:58:36.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T06:58:37.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:58:37.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:37.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:37.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:37.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:37.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:37.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:37.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:37.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:58:37.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:37.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:37.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:38.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:58:38.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:58:38.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:58:38.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:58:38.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:58:38.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:58:38.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:58:38.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:58:38.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:58:38.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:58:38.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:58:38.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T06:58:38.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:58:38.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T06:58:39.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:58:39.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:39.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:39.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:39.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:39.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:39.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:39.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:39.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:58:39.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:39.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:39.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:40.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:58:40.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:58:40.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:58:40.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:58:40.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:58:40.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:58:40.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:58:40.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:58:40.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:58:41.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:58:41.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:58:41.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T06:58:41.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:58:41.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T06:58:42.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:58:42.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:42.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:42.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:42.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:42.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:42.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:42.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:43.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:58:43.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:43.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:43.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:43.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:58:43.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:58:43.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:58:43.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:58:43.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:58:43.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:58:43.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:58:43.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:58:43.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:58:44.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:58:44.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:58:44.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T06:58:44.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:58:44.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T06:58:47.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:58:47.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:47.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:47.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:47.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:47.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:47.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:47.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:47.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:58:47.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:47.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:47.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:48.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:58:48.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:58:48.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:58:48.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:58:48.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:58:48.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:58:48.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:58:48.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:58:48.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:58:48.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:58:48.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:58:48.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T06:58:48.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:58:48.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T06:58:55.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:58:55.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:58:55.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:58:55.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:58:55.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:58:55.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:58:55.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:58:55.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:58:55.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:58:55.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:58:55.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:58:55.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:58:56.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:58:56.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:58:56.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:58:56.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T06:58:56.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:58:56.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T06:59:04.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:04.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:04.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:04.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:04.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:04.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:04.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:04.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:04.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:04.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:04.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:04.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:05.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:05.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:05.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:05.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:05.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:05.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:05.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:05.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:05.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:05.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:05.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:05.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T06:59:05.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T06:59:06.085 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T06:59:06.085 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "6f5649a0-1fce-4b53-b5d4-028beecb4bc2", 2024-04-25T06:59:06.085 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T06:59:06.085 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-04-25T06:59:06.087651+0000", 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "OSDMAP_FLAGS": { 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 17 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.086 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "OSD_DOWN": { 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 osds down", 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "osd.1 (root=default,host=smithi088) is down" 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.087 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:59:06.088 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.089 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:55:44.257576Z", 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T06:55:44.257576Z", 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-04-25T06:59:06.090 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-04-25T06:59:06.091 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.092 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115", 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-04-25T06:59:06.093 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 40, 2024-04-25T06:59:06.094 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T06:55:45.202302+0000", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:58:28.949962+0000", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-04-25T06:58:26.088801+0000", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-04-25T06:58:28.465011+0000", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5800384, 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "nodown", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "noin", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "noout", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-04-25T06:59:06.095 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 9, 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 3, 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-04-25T06:59:06.096 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "test", 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T06:57:30.543990+0000", 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 1, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool", 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T06:59:06.097 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 1, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 1, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 1, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 1, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 1, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T06:59:06.098 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T06:59:06.099 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T06:59:06.099 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.099 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "28", 2024-04-25T06:59:06.099 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T06:59:06.099 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T06:59:06.099 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T06:59:06.099 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T06:59:06.099 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 0, 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 0, 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T06:59:06.100 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T06:59:06.101 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {}, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 2.9900000095367432, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 2.9900000095367432, 2024-04-25T06:59:06.102 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 2, 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 2, 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:59:06.103 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6", 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 5, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 28, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 35, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 39, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 29, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.104 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.105 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.106 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6808", 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6809", 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.107 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6803/4091415756", 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6805/4091415756", 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6809/4091415756", 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6807/4091415756", 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T06:59:06.108 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "2357bb75-0d21-45fe-8b0a-60ccd011d618", 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 0, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 9, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 22, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 30, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:59:06.109 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.110 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.111 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.112 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:06.113 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.114 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.114 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.114 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:59:06.114 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.114 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.114 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.114 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T06:59:06.114 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.115 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/4024521766", 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/4024521766", 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4024521766", 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4024521766", 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: "exists" 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.116 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:59:06.117 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "e77a9999-9ad8-4555-a6ca-57b80790a230", 2024-04-25T06:59:06.117 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T06:59:06.117 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:59:06.117 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:59:06.117 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:59:06.117 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 14, 2024-04-25T06:59:06.117 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 28, 2024-04-25T06:59:06.117 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 37, 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 0, 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 31, 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T06:59:06.118 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.119 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.120 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.121 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T06:59:06.122 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:06.122 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.122 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.122 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.122 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T06:59:06.122 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:06.123 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/1330652640", 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/1330652640", 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1330652640", 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1330652640", 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.124 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.125 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-04-25T06:59:06.125 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.125 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:59:06.125 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:58:15.496692+0000", 2024-04-25T06:59:06.125 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:59:06.125 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:59:06.125 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:56:18.109921+0000", 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:58:16.021556+0000", 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:59:06.126 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:56:51.614962+0000", 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:58:16.587041+0000", 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:57:26.092048+0000", 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:59:06.127 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-04-25T06:59:06.128 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [], 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [], 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.129 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6804/4091415756,v1:127.0.0.1:6805/4091415756]", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:59:06.130 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:55:51.666950Z", 2024-04-25T06:59:06.131 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6802/4091415756,v1:127.0.0.1:6803/4091415756]", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/4091415756,v1:127.0.0.1:6809/4091415756]", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/4091415756,v1:127.0.0.1:6807/4091415756]", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:59:06.132 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:59:06.133 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/4024521766,v1:127.0.0.1:6813/4024521766]", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T06:59:06.134 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:56:24.576262Z", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:59:06.135 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/4024521766,v1:127.0.0.1:6811/4024521766]", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4024521766,v1:127.0.0.1:6817/4024521766]", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4024521766,v1:127.0.0.1:6815/4024521766]", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:59:06.136 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/1330652640,v1:127.0.0.1:6813/1330652640]", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:59:06.137 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:59:06.138 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:56:59.343178Z", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/1330652640,v1:127.0.0.1:6811/1330652640]", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1330652640,v1:127.0.0.1:6817/1330652640]", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1330652640,v1:127.0.0.1:6815/1330652640]", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:59:06.139 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-04-25T06:59:06.140 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [] 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 35 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 37 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.141 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 40, 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.1", 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:59:06.142 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.2", 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.143 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-04-25T06:59:06.144 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-04-25T06:59:06.145 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.146 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.147 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:59:06.148 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:59:06.149 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.150 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:59:06.151 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-04-25T06:59:06.152 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.153 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-04-25T06:59:06.154 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 0, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-04-25T06:59:06.155 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-04-25T06:59:06.156 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-04-25T06:59:06.157 INFO:tasks.workunit.client.0.smithi088.stderr:report 3091440083 2024-04-25T06:59:06.157 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-04-25T06:59:06.157 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-04-25T06:59:06.157 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-04-25T06:59:06.157 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T06:59:06.157 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-04-25T06:59:06.157 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T06:59:06.157 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-04-25T06:59:06.158 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 5, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 3 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 1, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 3, 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-04-25T06:59:06.159 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 61700, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 100, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 300, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 100, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 100, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T06:59:06.160 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 100, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 100, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T06:59:06.161 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.162 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 16, 2024-04-25T06:59:06.163 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 16, 2024-04-25T06:59:06.164 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 2, 2024-04-25T06:59:06.164 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 2, 2024-04-25T06:59:06.164 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-04-25T06:59:06.164 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.164 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-04-25T06:59:06.164 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 83596, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2080, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81461, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314489204, 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T06:59:06.165 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036944896, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2129920, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 455064, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83416932 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T06:59:06.166 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T06:59:06.167 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 83596, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2080, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81461, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314489204, 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T06:59:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036944896, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2129920, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 455064, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83416932 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T06:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 61700, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 100, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 300, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 100, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 100, 2024-04-25T06:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 100, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 100, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T06:59:06.172 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T06:59:06.173 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1220608, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 184566, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T06:59:06.174 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 16, 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 16, 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 2, 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 2, 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 3 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 150323855371 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:59:06.175 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 38654705692 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 158913789963 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 1 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-04-25T06:59:06.176 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 1, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T06:59:06.177 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 0, 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 5, 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:58:18.665598+0000", 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-04-25T06:59:06.178 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 89, 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 369, 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 507, 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:06.179 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T06:59:06.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T06:59:06.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-04-25T06:59:06.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-04-25T06:59:06.180 INFO:tasks.workunit.client.0.smithi088.stdout:writing non-divergent object existing_4 2024-04-25T06:59:06.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-04-25T06:59:06.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:59:06.506 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:59:06.507 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T06:58:29.158313+0000 28'100 40:128 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:45:18.553052+0000 0 0 2024-04-25T06:59:06.507 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:59:06.507 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:59:06.507 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:59:06.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T06:59:06.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T06:59:06.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-04-25T06:59:06.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T06:59:06.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T06:59:06.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T06:59:06.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T06:59:06.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T06:59:06.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T06:59:06.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T06:59:06.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T06:59:06.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T06:59:06.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T06:59:06.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T06:59:06.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T06:59:06.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T06:59:06.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T06:59:06.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T06:59:07.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T06:59:07.105 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T06:59:07.106 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T06:59:07.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T06:59:07.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:59:07.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T06:59:07.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855373 2024-04-25T06:59:07.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855373 2024-04-25T06:59:07.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855373' 2024-04-25T06:59:07.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:59:07.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T06:59:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T06:59:07.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T06:59:07.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T06:59:07.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T06:59:07.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T06:59:07.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T06:59:07.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789966 2024-04-25T06:59:07.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789966 2024-04-25T06:59:07.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855373 2-158913789966' 2024-04-25T06:59:07.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:59:07.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855373 2024-04-25T06:59:07.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:59:07.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T06:59:07.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855373 2024-04-25T06:59:07.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:59:07.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855373 2024-04-25T06:59:07.572 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 150323855373 2024-04-25T06:59:07.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855373' 2024-04-25T06:59:07.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:59:07.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855371 -lt 150323855373 2024-04-25T06:59:07.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T06:59:08.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T06:59:08.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T06:59:09.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855374 -lt 150323855373 2024-04-25T06:59:09.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T06:59:09.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T06:59:09.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-158913789966 2024-04-25T06:59:09.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T06:59:09.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-158913789966 2024-04-25T06:59:09.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T06:59:09.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789966 2024-04-25T06:59:09.455 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 158913789966 2024-04-25T06:59:09.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 158913789966' 2024-04-25T06:59:09.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T06:59:09.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789966 -lt 158913789966 2024-04-25T06:59:09.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T06:59:09.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:09.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:10.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T06:59:10.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:10.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:10.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:10.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:10.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:10.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:10.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:10.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:10.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:10.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:10.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:11.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:11.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T06:59:11.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T06:59:11.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T06:59:11.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:59:11.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T06:59:11.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:11.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:11.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:11.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:11.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:11.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:11.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:11.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:11.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:11.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:11.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:11.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:12.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:12.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:12.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:12.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:12.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:12.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:12.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:12.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:12.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:13.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:13.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:13.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T06:59:13.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:59:13.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T06:59:13.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:13.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:13.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:13.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:13.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:13.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:13.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:13.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:13.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:13.630 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:13.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:13.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:14.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:14.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:14.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:14.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:14.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:14.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:14.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:14.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:14.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:14.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:14.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:14.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T06:59:14.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:59:14.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T06:59:15.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:15.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:15.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:15.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:15.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:15.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:15.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:15.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:15.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:15.523 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:15.523 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:15.523 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:16.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:16.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:16.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:16.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:16.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:16.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:16.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:16.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:16.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:16.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:16.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:16.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T06:59:16.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:59:16.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T06:59:17.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:17.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:17.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:17.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:17.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:17.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:17.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:17.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:17.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:17.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:17.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:17.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:18.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:18.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:18.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:18.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:18.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:18.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:18.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:18.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:18.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:18.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:18.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:18.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T06:59:18.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:59:18.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T06:59:20.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:20.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:20.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:20.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:20.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:20.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:20.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:20.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:20.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:20.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:20.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:20.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:21.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:21.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:21.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:21.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:21.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:21.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:21.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:21.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:21.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:21.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:21.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:21.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T06:59:21.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:59:21.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T06:59:25.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:25.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:25.201 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:25.201 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:25.201 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:25.201 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:25.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:25.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:25.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:25.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:25.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:25.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:26.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:26.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:26.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:26.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:26.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T06:59:26.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:59:26.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T06:59:33.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:33.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:33.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:33.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:33.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:33.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:33.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:33.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:33.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:33.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:33.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:33.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:34.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:34.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:34.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:34.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:34.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:34.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:34.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:34.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:34.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:34.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:34.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:34.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T06:59:34.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T06:59:34.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T06:59:41.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T06:59:41.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T06:59:41.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T06:59:41.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T06:59:41.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T06:59:41.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T06:59:41.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T06:59:41.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T06:59:42.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T06:59:42.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T06:59:42.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T06:59:42.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T06:59:42.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T06:59:42.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T06:59:42.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T06:59:42.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T06:59:42.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T06:59:42.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T06:59:42.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T06:59:42.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T06:59:42.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T06:59:43.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T06:59:43.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T06:59:43.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T06:59:43.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "6f5649a0-1fce-4b53-b5d4-028beecb4bc2", 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-04-25T06:59:43.974234+0000", 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-04-25T06:59:43.973 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: "OSDMAP_FLAGS": { 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 17 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.974 INFO:tasks.workunit.client.0.smithi088.stdout: "OSD_DOWN": { 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 osds down", 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "osd.1 (root=default,host=smithi088) is down" 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:59:43.975 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 2 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T06:59:43.976 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.977 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:55:44.257576Z", 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T06:55:44.257576Z", 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-04-25T06:59:43.978 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-04-25T06:59:43.979 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115", 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.980 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 40, 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T06:59:43.981 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T06:55:45.202302+0000", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:58:28.949962+0000", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-04-25T06:58:26.088801+0000", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-04-25T06:58:28.465011+0000", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5800384, 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "nodown", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "noin", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "noout", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-04-25T06:59:43.982 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 9, 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 3, 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T06:59:43.983 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "test", 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T06:57:30.543990+0000", 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 1, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool", 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-04-25T06:59:43.984 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 1, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 1, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 1, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 1, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 1, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "28", 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T06:59:43.985 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 0, 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 0, 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T06:59:43.986 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T06:59:43.987 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {}, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 2.9900000095367432, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 2.9900000095367432, 2024-04-25T06:59:43.988 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 2, 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 2, 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6", 2024-04-25T06:59:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 5, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 28, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 35, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 39, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 29, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-04-25T06:59:43.990 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.991 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6808", 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.992 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6809", 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:43.993 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4091415756 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6803/4091415756", 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6805/4091415756", 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6809/4091415756", 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6807/4091415756", 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.994 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "2357bb75-0d21-45fe-8b0a-60ccd011d618", 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 0, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 9, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 22, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 30, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.995 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.996 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.997 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.998 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 4024521766 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/4024521766", 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/4024521766", 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4024521766", 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4024521766", 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:59:43.999 INFO:tasks.workunit.client.0.smithi088.stdout: "exists" 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "e77a9999-9ad8-4555-a6ca-57b80790a230", 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 14, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 28, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 37, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 0, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 31, 2024-04-25T06:59:44.000 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.001 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T06:59:44.002 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.003 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1330652640 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/1330652640", 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/1330652640", 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1330652640", 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1330652640", 2024-04-25T06:59:44.004 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:58:15.496692+0000", 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:56:18.109921+0000", 2024-04-25T06:59:44.005 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:58:16.021556+0000", 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:56:51.614962+0000", 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T06:58:16.587041+0000", 2024-04-25T06:59:44.006 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T06:57:26.092048+0000", 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-04-25T06:59:44.007 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [], 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [], 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-04-25T06:59:44.008 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6804/4091415756,v1:127.0.0.1:6805/4091415756]", 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:59:44.009 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:59:44.010 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:55:51.666950Z", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6802/4091415756,v1:127.0.0.1:6803/4091415756]", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/4091415756,v1:127.0.0.1:6809/4091415756]", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/4091415756,v1:127.0.0.1:6807/4091415756]", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:59:44.011 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:59:44.012 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/4024521766,v1:127.0.0.1:6813/4024521766]", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:59:44.013 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:56:24.576262Z", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:59:44.014 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/4024521766,v1:127.0.0.1:6811/4024521766]", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4024521766,v1:127.0.0.1:6817/4024521766]", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4024521766,v1:127.0.0.1:6815/4024521766]", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T06:59:44.015 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/1330652640,v1:127.0.0.1:6813/1330652640]", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T06:59:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T06:59:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T06:56:59.343178Z", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/1330652640,v1:127.0.0.1:6811/1330652640]", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1330652640,v1:127.0.0.1:6817/1330652640]", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1330652640,v1:127.0.0.1:6815/1330652640]", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T06:59:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-04-25T06:59:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [] 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 35 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 37 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 40, 2024-04-25T06:59:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.1", 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.2", 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T06:59:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.022 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.023 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.024 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:59:44.025 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:59:44.026 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:59:44.027 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T06:59:44.028 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T06:59:44.029 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.030 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T06:59:44.031 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-04-25T06:59:44.032 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 0, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-04-25T06:59:44.033 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-04-25T06:59:44.034 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-04-25T06:59:44.035 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 5, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 3 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 1, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 3, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 62050, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 100, 2024-04-25T06:59:44.036 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 300, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 100, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 100, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 101, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 101, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T06:59:44.037 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T06:59:44.038 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T06:59:44.039 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 17, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 17, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 2, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 2, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T06:59:44.040 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 83596, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2080, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81461, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314489204, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036944896, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2129920, 2024-04-25T06:59:44.041 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 455764, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83416932 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.042 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T06:59:44.043 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 83596, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2080, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81461, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314489204, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036944896, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:59:44.044 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2129920, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 455764, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83416932 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T06:59:44.045 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T06:59:44.046 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 62050, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 100, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 300, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 100, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 100, 2024-04-25T06:59:44.047 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 101, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 101, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T06:59:44.048 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T06:59:44.049 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1220608, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 185266, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 17, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 17, 2024-04-25T06:59:44.050 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 2, 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 2, 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 3 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 150323855380 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 38654705692 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.051 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 158913789973 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 1 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T06:59:44.052 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 1, 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 0, 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.053 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 7, 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T06:59:28.685500+0000", 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 108, 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 437, 2024-04-25T06:59:44.054 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-04-25T06:59:44.055 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T06:59:44.055 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 594, 2024-04-25T06:59:44.055 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-04-25T06:59:44.055 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-04-25T06:59:44.055 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T06:59:44.055 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T06:59:44.055 INFO:tasks.workunit.client.0.smithi088.stderr:report 2761356614 2024-04-25T06:59:44.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T06:59:44.056 INFO:tasks.workunit.client.0.smithi088.stdout:delay recovery 2024-04-25T06:59:44.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-04-25T06:59:44.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:59:44.388 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:59:44.388 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T06:58:29.158313+0000 40'101 40:129 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:45:18.553052+0000 0 0 2024-04-25T06:59:44.388 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:59:44.388 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:59:44.388 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:59:44.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:59:44.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-04-25T06:59:44.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T06:59:44.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T06:59:44.404 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:59:44.404 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:59:44.404 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:59:44.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.0.asok 2024-04-25T06:59:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-04-25T06:59:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T06:59:44.516 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T06:59:44.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-04-25T06:59:44.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-04-25T06:59:44.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T06:59:44.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T06:59:44.531 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:59:44.531 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:59:44.531 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:59:44.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.2.asok 2024-04-25T06:59:44.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-04-25T06:59:44.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T06:59:44.640 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T06:59:44.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-04-25T06:59:44.651 INFO:tasks.workunit.client.0.smithi088.stdout:revive divergent 1 2024-04-25T06:59:44.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:59:45.055 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:59:45.055 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T06:58:29.158313+0000 40'101 40:129 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:45:18.553052+0000 0 0 2024-04-25T06:59:45.056 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:59:45.056 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:59:45.056 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:59:45.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-04-25T06:59:45.748 INFO:tasks.workunit.client.0.smithi088.stderr:noup is set 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T06:59:45.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T06:59:45.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T06:59:45.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T06:59:45.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T06:59:45.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T06:59:45.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T06:59:45.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T06:59:45.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T06:59:45.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T06:59:45.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T06:59:45.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T06:59:45.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T06:59:45.774 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T06:59:45.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T06:59:45.775 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T06:59:45.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T06:59:45.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T06:59:45.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T06:59:45.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T06:59:45.810 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:59:45.811+0000 7fb513e3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:59:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:59:45.811+0000 7fb513e3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:59:45.814 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:59:45.815+0000 7fb513e3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T06:59:46.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T06:59:46.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-04-25T06:59:46.384 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:59:46.383+0000 7fb513e3c7c0 -1 Falling back to public interface 2024-04-25T06:59:49.241 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T06:59:49.239+0000 7fb513e3c7c0 -1 osd.1 28 log_to_monitors true 2024-04-25T06:59:51.220 INFO:tasks.workunit.client.0.smithi088.stdout:delay recovery divergent 2024-04-25T06:59:51.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-04-25T06:59:51.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:59:51.626 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:59:51.626 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T06:58:29.158313+0000 40'101 42:132 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:45:18.553052+0000 0 0 2024-04-25T06:59:51.627 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:59:51.627 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:59:51.627 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:59:51.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-04-25T06:59:51.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T06:59:51.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T06:59:51.643 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T06:59:51.643 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T06:59:51.643 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T06:59:51.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.1.asok 2024-04-25T06:59:51.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-04-25T06:59:51.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T06:59:51.755 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T06:59:51.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-04-25T06:59:52.377 INFO:tasks.workunit.client.0.smithi088.stderr:noup is unset 2024-04-25T06:59:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-04-25T06:59:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:59:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T06:59:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:59:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:59:52.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:59:52.398 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:59:52.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:59:52.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:59:52.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T06:59:52.964 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 35 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/4091415756,v1:127.0.0.1:6803/4091415756] [v2:127.0.0.1:6804/4091415756,v1:127.0.0.1:6805/4091415756] exists,up ffc17e0e-a1bf-46ad-8d47-295f31a8a4d6 2024-04-25T06:59:52.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:59:52.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:59:52.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:59:52.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-04-25T06:59:52.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:59:52.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T06:59:52.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:59:52.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:59:52.966 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:59:52.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:59:52.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:59:52.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T06:59:52.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:59:53.397 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/345125639,v1:127.0.0.1:6819/345125639] [v2:127.0.0.1:6820/345125639,v1:127.0.0.1:6821/345125639] exists,up 2357bb75-0d21-45fe-8b0a-60ccd011d618 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T06:59:53.399 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T06:59:53.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T06:59:53.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T06:59:53.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T06:59:53.879 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [14,28) [v2:127.0.0.1:6810/1330652640,v1:127.0.0.1:6811/1330652640] [v2:127.0.0.1:6812/1330652640,v1:127.0.0.1:6813/1330652640] exists,up e77a9999-9ad8-4555-a6ca-57b80790a230 2024-04-25T06:59:53.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T06:59:53.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T06:59:53.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T06:59:53.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:59:54.301 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:59:54.301 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T06:58:29.158313+0000 40'101 42:132 [0,2] 0 [0,2] 0 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:45:18.553052+0000 0 0 2024-04-25T06:59:54.301 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:59:54.301 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:59:54.302 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:59:54.318 INFO:tasks.workunit.client.0.smithi088.stdout:wait for peering 2024-04-25T06:59:54.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-04-25T06:59:54.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:59:54.701 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:59:54.701 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-04-25T06:59:53.024020+0000 40'101 46:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 no scrub is scheduled 0 0 2024-04-25T06:59:54.701 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:59:54.701 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:59:54.701 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:59:54.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-04-25T06:59:54.837 INFO:tasks.workunit.client.0.smithi088.stdout:killing divergent 1 2024-04-25T06:59:54.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-04-25T06:59:54.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-04-25T06:59:55.221 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T06:59:55.222 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-04-25T06:59:53.024020+0000 40'101 46:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 no scrub is scheduled 0 0 2024-04-25T06:59:55.222 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T06:59:55.222 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T06:59:55.222 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T06:59:55.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T06:59:55.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T06:59:55.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T06:59:55.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T06:59:55.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T06:59:55.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T06:59:55.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T06:59:55.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.26017.out 2024-04-25T06:59:55.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.26017.out 2024-04-25T06:59:55.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T06:59:55.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T06:59:55.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T06:59:55.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T06:59:55.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T06:59:55.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.26017.out 2024-04-25T06:59:56.407 INFO:tasks.workunit.client.0.smithi088.stderr:Exporting 1.0 info 1.0( v 46'102 lc 0'0 (28'90,46'102] local-lis/les=45/46 n=101 ec=22/22 lis/c=45/22 les/c/f=46/23/0 sis=45) 2024-04-25T06:59:56.409 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:023dd3d2:::existing_12:head# 2024-04-25T06:59:56.418 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:04c40a0c:::existing_93:head# 2024-04-25T06:59:56.419 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:064ff5fb:::existing_36:head# 2024-04-25T06:59:56.420 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:091f9f1d:::existing_94:head# 2024-04-25T06:59:56.421 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0a865a93:::existing_54:head# 2024-04-25T06:59:56.421 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0fed513c:::existing_41:head# 2024-04-25T06:59:56.422 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-04-25T06:59:56.422 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:169157fa:::existing_81:head# 2024-04-25T06:59:56.422 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:17731382:::existing_58:head# 2024-04-25T06:59:56.423 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:18a45a39:::existing_43:head# 2024-04-25T06:59:56.423 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1c160f01:::existing_66:head# 2024-04-25T06:59:56.423 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1dd6c249:::existing_97:head# 2024-04-25T06:59:56.424 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:227e0c77:::existing_7:head# 2024-04-25T06:59:56.435 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:23e1b063:::existing_59:head# 2024-04-25T06:59:56.436 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:28566edb:::existing_78:head# 2024-04-25T06:59:56.436 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2a101182:::existing_20:head# 2024-04-25T06:59:56.437 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2d1519da:::existing_96:head# 2024-04-25T06:59:56.437 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3078b87a:::existing_86:head# 2024-04-25T06:59:56.439 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:30bde01b:::existing_15:head# 2024-04-25T06:59:56.439 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:317a7e19:::existing_56:head# 2024-04-25T06:59:56.439 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-04-25T06:59:56.440 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3e16d84b:::existing_48:head# 2024-04-25T06:59:56.440 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:45fe8203:::existing_75:head# 2024-04-25T06:59:56.440 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-04-25T06:59:56.440 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5298dcdb:::existing_57:head# 2024-04-25T06:59:56.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:52ebaa44:::existing_32:head# 2024-04-25T06:59:56.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:53588912:::existing_9:head# 2024-04-25T06:59:56.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-04-25T06:59:56.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5da0255c:::existing_13:head# 2024-04-25T06:59:56.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5da2dab8:::existing_53:head# 2024-04-25T06:59:56.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5ee3514b:::existing_30:head# 2024-04-25T06:59:56.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:602f83fe:::foo:head# 2024-04-25T06:59:56.443 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:60426cc7:::existing_62:head# 2024-04-25T06:59:56.443 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:628918ca:::existing_79:head# 2024-04-25T06:59:56.443 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-04-25T06:59:56.443 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:67e1780f:::existing_27:head# 2024-04-25T06:59:56.445 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:68143ca1:::existing_71:head# 2024-04-25T06:59:56.446 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6bac9509:::existing_29:head# 2024-04-25T06:59:56.446 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6bbec798:::existing_45:head# 2024-04-25T06:59:56.447 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6cb23a40:::existing_65:head# 2024-04-25T06:59:56.448 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7055ec5e:::existing_83:head# 2024-04-25T06:59:56.448 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:70a54e2b:::existing_87:head# 2024-04-25T06:59:56.448 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:750ace21:::existing_19:head# 2024-04-25T06:59:56.448 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:78623b1d:::existing_6:head# 2024-04-25T06:59:56.451 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:79881152:::existing_63:head# 2024-04-25T06:59:56.452 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-04-25T06:59:56.453 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81490677:::existing_76:head# 2024-04-25T06:59:56.454 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-04-25T06:59:56.454 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-04-25T06:59:56.454 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-04-25T06:59:56.454 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:923c5d4d:::existing_5:head# 2024-04-25T06:59:56.455 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:92731e71:::existing_89:head# 2024-04-25T06:59:56.455 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:928205cf:::existing_82:head# 2024-04-25T06:59:56.455 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:93be2267:::existing_24:head# 2024-04-25T06:59:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:99ceda2c:::existing_44:head# 2024-04-25T06:59:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9dd917ee:::existing_26:head# 2024-04-25T06:59:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a28805a6:::existing_72:head# 2024-04-25T06:59:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a294a8f6:::existing_16:head# 2024-04-25T06:59:56.457 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a2f5473a:::existing_42:head# 2024-04-25T06:59:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a504a533:::existing_90:head# 2024-04-25T06:59:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a691337e:::existing_61:head# 2024-04-25T06:59:56.458 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-04-25T06:59:56.459 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a9fde86e:::existing_18:head# 2024-04-25T06:59:56.459 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:aa309faa:::existing_68:head# 2024-04-25T06:59:56.459 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-04-25T06:59:56.460 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b279b1fe:::existing_10:head# 2024-04-25T06:59:56.460 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b343e277:::existing_88:head# 2024-04-25T06:59:56.460 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b7130e90:::existing_80:head# 2024-04-25T06:59:56.460 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bc6f7309:::existing_70:head# 2024-04-25T06:59:56.461 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:be8489b7:::existing_38:head# 2024-04-25T06:59:56.462 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-04-25T06:59:56.462 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bebec877:::existing_74:head# 2024-04-25T06:59:56.462 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c04347d8:::existing_60:head# 2024-04-25T06:59:56.462 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c156033e:::existing_100:head# 2024-04-25T06:59:56.463 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-04-25T06:59:56.463 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-04-25T06:59:56.463 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-04-25T06:59:56.463 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-04-25T06:59:56.463 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ccb3865b:::existing_73:head# 2024-04-25T06:59:56.465 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cd84e19e:::existing_40:head# 2024-04-25T06:59:56.465 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-04-25T06:59:56.465 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d41a25c9:::existing_22:head# 2024-04-25T06:59:56.465 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d7f088da:::existing_91:head# 2024-04-25T06:59:56.465 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-04-25T06:59:56.465 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-04-25T06:59:56.466 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dd677bd3:::existing_35:head# 2024-04-25T06:59:56.466 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-04-25T06:59:56.466 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e0e22077:::existing_21:head# 2024-04-25T06:59:56.466 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e255b710:::existing_55:head# 2024-04-25T06:59:56.467 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:eff1d5db:::existing_14:head# 2024-04-25T06:59:56.467 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f1b156bf:::existing_23:head# 2024-04-25T06:59:56.467 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f5ce4199:::existing_34:head# 2024-04-25T06:59:56.467 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f6809456:::existing_11:head# 2024-04-25T06:59:56.468 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f699a2a9:::existing_67:head# 2024-04-25T06:59:56.468 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f9baac43:::existing_25:head# 2024-04-25T06:59:56.468 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fb76688e:::existing_51:head# 2024-04-25T06:59:56.469 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fbb92eec:::existing_69:head# 2024-04-25T06:59:56.469 INFO:tasks.workunit.client.0.smithi088.stderr:Export successful 2024-04-25T06:59:56.469 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T06:59:56.469 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T06:59:56.470 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T06:59:56.665 INFO:tasks.workunit.client.0.smithi088.stderr:Remove successful 2024-04-25T06:59:57.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.26017.out 2024-04-25T06:59:57.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T06:59:57.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T06:59:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T06:59:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T06:59:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T06:59:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.26017.out 2024-04-25T06:59:57.829 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e46 pg_nums {1={22=1}} deleted_pools ) 2024-04-25T06:59:57.830 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.0 2024-04-25T06:59:57.830 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:023dd3d2:::existing_12:head# 2024-04-25T06:59:57.830 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:57.893 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:04c40a0c:::existing_93:head# 2024-04-25T06:59:57.894 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:57.927 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:064ff5fb:::existing_36:head# 2024-04-25T06:59:57.927 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:57.960 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:091f9f1d:::existing_94:head# 2024-04-25T06:59:57.960 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:57.994 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a865a93:::existing_54:head# 2024-04-25T06:59:57.994 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.027 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0fed513c:::existing_41:head# 2024-04-25T06:59:58.027 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.061 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-04-25T06:59:58.061 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.094 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:169157fa:::existing_81:head# 2024-04-25T06:59:58.094 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.127 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17731382:::existing_58:head# 2024-04-25T06:59:58.128 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.161 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:18a45a39:::existing_43:head# 2024-04-25T06:59:58.161 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.194 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1c160f01:::existing_66:head# 2024-04-25T06:59:58.195 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.228 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1dd6c249:::existing_97:head# 2024-04-25T06:59:58.228 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.261 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:227e0c77:::existing_7:head# 2024-04-25T06:59:58.261 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.295 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:23e1b063:::existing_59:head# 2024-04-25T06:59:58.295 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.328 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:28566edb:::existing_78:head# 2024-04-25T06:59:58.328 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.362 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2a101182:::existing_20:head# 2024-04-25T06:59:58.362 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.395 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2d1519da:::existing_96:head# 2024-04-25T06:59:58.396 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.429 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3078b87a:::existing_86:head# 2024-04-25T06:59:58.429 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.462 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:30bde01b:::existing_15:head# 2024-04-25T06:59:58.462 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.495 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:317a7e19:::existing_56:head# 2024-04-25T06:59:58.496 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.529 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-04-25T06:59:58.529 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.562 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3e16d84b:::existing_48:head# 2024-04-25T06:59:58.563 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.596 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:45fe8203:::existing_75:head# 2024-04-25T06:59:58.596 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.629 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-04-25T06:59:58.630 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.663 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5298dcdb:::existing_57:head# 2024-04-25T06:59:58.663 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.696 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:52ebaa44:::existing_32:head# 2024-04-25T06:59:58.696 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.730 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:53588912:::existing_9:head# 2024-04-25T06:59:58.730 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.772 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-04-25T06:59:58.772 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.839 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5da0255c:::existing_13:head# 2024-04-25T06:59:58.839 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.864 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5da2dab8:::existing_53:head# 2024-04-25T06:59:58.864 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.889 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5ee3514b:::existing_30:head# 2024-04-25T06:59:58.889 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.914 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:602f83fe:::foo:head# 2024-04-25T06:59:58.914 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.939 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:60426cc7:::existing_62:head# 2024-04-25T06:59:58.939 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.964 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:628918ca:::existing_79:head# 2024-04-25T06:59:58.964 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:58.989 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-04-25T06:59:58.989 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.014 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67e1780f:::existing_27:head# 2024-04-25T06:59:59.015 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.048 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:68143ca1:::existing_71:head# 2024-04-25T06:59:59.048 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.081 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6bac9509:::existing_29:head# 2024-04-25T06:59:59.082 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.115 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6bbec798:::existing_45:head# 2024-04-25T06:59:59.115 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.148 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6cb23a40:::existing_65:head# 2024-04-25T06:59:59.149 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.182 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7055ec5e:::existing_83:head# 2024-04-25T06:59:59.182 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.241 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:70a54e2b:::existing_87:head# 2024-04-25T06:59:59.241 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.308 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:750ace21:::existing_19:head# 2024-04-25T06:59:59.308 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.358 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:78623b1d:::existing_6:head# 2024-04-25T06:59:59.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.391 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:79881152:::existing_63:head# 2024-04-25T06:59:59.391 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.425 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-04-25T06:59:59.425 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.458 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81490677:::existing_76:head# 2024-04-25T06:59:59.458 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.492 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-04-25T06:59:59.492 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.525 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-04-25T06:59:59.525 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.559 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-04-25T06:59:59.559 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.592 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:923c5d4d:::existing_5:head# 2024-04-25T06:59:59.592 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.626 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92731e71:::existing_89:head# 2024-04-25T06:59:59.626 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.659 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:928205cf:::existing_82:head# 2024-04-25T06:59:59.659 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.693 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:93be2267:::existing_24:head# 2024-04-25T06:59:59.693 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.743 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:99ceda2c:::existing_44:head# 2024-04-25T06:59:59.743 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.776 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9dd917ee:::existing_26:head# 2024-04-25T06:59:59.776 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.810 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a28805a6:::existing_72:head# 2024-04-25T06:59:59.810 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.843 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a294a8f6:::existing_16:head# 2024-04-25T06:59:59.843 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.876 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a2f5473a:::existing_42:head# 2024-04-25T06:59:59.877 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.910 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a504a533:::existing_90:head# 2024-04-25T06:59:59.910 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.943 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a691337e:::existing_61:head# 2024-04-25T06:59:59.944 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T06:59:59.977 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-04-25T06:59:59.977 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.010 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a9fde86e:::existing_18:head# 2024-04-25T07:00:00.010 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.052 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:aa309faa:::existing_68:head# 2024-04-25T07:00:00.052 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.094 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-04-25T07:00:00.094 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.152 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b279b1fe:::existing_10:head# 2024-04-25T07:00:00.153 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.194 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b343e277:::existing_88:head# 2024-04-25T07:00:00.194 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.228 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b7130e90:::existing_80:head# 2024-04-25T07:00:00.228 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.261 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bc6f7309:::existing_70:head# 2024-04-25T07:00:00.261 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.295 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:be8489b7:::existing_38:head# 2024-04-25T07:00:00.295 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.328 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-04-25T07:00:00.328 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.362 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bebec877:::existing_74:head# 2024-04-25T07:00:00.362 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.395 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c04347d8:::existing_60:head# 2024-04-25T07:00:00.395 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.428 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c156033e:::existing_100:head# 2024-04-25T07:00:00.429 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.462 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-04-25T07:00:00.462 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.495 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-04-25T07:00:00.495 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.529 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-04-25T07:00:00.529 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.562 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-04-25T07:00:00.562 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.596 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ccb3865b:::existing_73:head# 2024-04-25T07:00:00.596 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.629 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cd84e19e:::existing_40:head# 2024-04-25T07:00:00.629 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.663 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-04-25T07:00:00.663 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.696 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d41a25c9:::existing_22:head# 2024-04-25T07:00:00.696 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.729 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d7f088da:::existing_91:head# 2024-04-25T07:00:00.730 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.771 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:00:00.772 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.838 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-04-25T07:00:00.839 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.872 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd677bd3:::existing_35:head# 2024-04-25T07:00:00.872 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.905 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-04-25T07:00:00.906 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.939 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0e22077:::existing_21:head# 2024-04-25T07:00:00.939 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:00.972 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e255b710:::existing_55:head# 2024-04-25T07:00:00.973 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.006 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:eff1d5db:::existing_14:head# 2024-04-25T07:00:01.006 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.031 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f1b156bf:::existing_23:head# 2024-04-25T07:00:01.031 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.056 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f5ce4199:::existing_34:head# 2024-04-25T07:00:01.056 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.081 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f6809456:::existing_11:head# 2024-04-25T07:00:01.081 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.106 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f699a2a9:::existing_67:head# 2024-04-25T07:00:01.107 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.131 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f9baac43:::existing_25:head# 2024-04-25T07:00:01.132 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.157 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fb76688e:::existing_51:head# 2024-04-25T07:00:01.157 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.182 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fbb92eec:::existing_69:head# 2024-04-25T07:00:01.182 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:00:01.232 INFO:tasks.workunit.client.0.smithi088.stdout:write_pg epoch 46 info 1.0( v 46'102 lc 0'0 (28'90,46'102] local-lis/les=45/46 n=101 ec=22/22 lis/c=45/22 les/c/f=46/23/0 sis=45) 2024-04-25T07:00:01.282 INFO:tasks.workunit.client.0.smithi088.stdout:Import successful 2024-04-25T07:00:01.829 INFO:tasks.workunit.client.0.smithi088.stdout:reviving divergent 1 2024-04-25T07:00:01.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-04-25T07:00:01.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:00:02.246 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:00:02.246 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-04-25T06:59:53.024020+0000 40'101 46:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:00:02.246 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:00:02.246 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:00:02.246 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:00:02.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:00:02.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:00:02.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:00:02.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:00:02.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:00:02.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:00:02.266 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:00:02.266 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:02.266 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:02.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:00:02.267 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:00:02.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:00:02.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:00:02.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:00:02.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:00:02.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:00:02.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:00:02.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:00:02.306 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:02.307+0000 7fa1365707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:02.308 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:02.307+0000 7fa1365707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:02.309 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:02.311+0000 7fa1365707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:02.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:00:02.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:00:02.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:00:02.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:00:02.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:00:02.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:02.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:00:02.721 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:00:02.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:02.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:00:02.847 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:02.847+0000 7fa1365707c0 -1 Falling back to public interface 2024-04-25T07:00:03.119 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/345125639,v1:127.0.0.1:6819/345125639] [v2:127.0.0.1:6820/345125639,v1:127.0.0.1:6821/345125639] exists,up 2357bb75-0d21-45fe-8b0a-60ccd011d618 2024-04-25T07:00:03.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:00:03.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:00:03.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:00:03.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-04-25T07:00:03.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:00:03.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:00:03.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:00:03.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:00:03.121 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:00:03.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:03.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:00:03.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:03.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:00:03.540 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/345125639,v1:127.0.0.1:6819/345125639] [v2:127.0.0.1:6820/345125639,v1:127.0.0.1:6821/345125639] exists,up 2357bb75-0d21-45fe-8b0a-60ccd011d618 2024-04-25T07:00:03.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:00:03.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:00:03.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:00:03.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-04-25T07:00:05.778 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:05.779+0000 7fa1365707c0 -1 osd.1 46 log_to_monitors true 2024-04-25T07:00:23.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-04-25T07:00:23.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:00:23.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:00:23.545 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:00:23.545 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:23.545 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:23.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.1.asok 2024-04-25T07:00:23.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:00:23.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T07:00:23.662 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:00:23.662 INFO:tasks.workunit.client.0.smithi088.stdout: "ops": [], 2024-04-25T07:00:23.663 INFO:tasks.workunit.client.0.smithi088.stdout: "num_ops": 0 2024-04-25T07:00:23.663 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:00:23.675 INFO:tasks.workunit.client.0.smithi088.stdout:allowing recovery 2024-04-25T07:00:23.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-04-25T07:00:23.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:00:24.077 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:00:24.077 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-04-25T07:00:07.473029+0000 46'102 50:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:14:26.892891+0000 0 0 2024-04-25T07:00:24.077 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:00:24.077 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:00:24.077 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:00:24.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:00:24.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:00:24.237 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:00:24.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:00:24.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:00:24.392 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:00:24.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:00:24.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:00:24.551 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:00:24.565 INFO:tasks.workunit.client.0.smithi088.stdout:reading divergent objects 2024-04-25T07:00:24.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-04-25T07:00:24.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:00:24.961 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:00:24.961 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-04-25T07:00:07.473029+0000 46'102 50:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T06:57:30.594277+0000 0'0 2024-04-25T06:57:30.594277+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:14:26.892891+0000 0 0 2024-04-25T07:00:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:00:24.961 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:00:24.961 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:00:24.977 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-04-25T07:00:24.978 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-04-25T07:00:24.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:00:24.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:00:25.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:00:25.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:00:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:00:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:00:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:00:25.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:00:25.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-04-25T07:00:25.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:00:25.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-04-25T07:00:25.145 INFO:tasks.workunit.client.0.smithi088.stderr:error getting test/newobject_1: (2) No such file or directory 2024-04-25T07:00:25.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:00:25.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-04-25T07:00:25.178 INFO:tasks.workunit.client.0.smithi088.stderr:error getting test/newobject_2: (2) No such file or directory 2024-04-25T07:00:25.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-04-25T07:00:25.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:00:25.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4631.0:1 2024-04-25T06:58:04.625342+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: keeping 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4631.0:1 2024-04-25T06:58:04.625342+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'106 last_divergent_update: 28'106 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: more recent entry found: 40'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4746.0:1 2024-04-25T06:59:06.565882+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4627.0:1 2024-04-25T06:58:04.617200+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: keeping 28'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4627.0:1 2024-04-25T06:58:04.617200+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'104 last_divergent_update: 28'104 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:00:25.186 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4628.0:1 2024-04-25T06:58:04.617641+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: keeping 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4628.0:1 2024-04-25T06:58:04.617641+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'105 last_divergent_update: 28'105 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4628.0:1 2024-04-25T06:58:04.617641+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4619.0:1 2024-04-25T06:58:04.604285+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: keeping 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4619.0:1 2024-04-25T06:58:04.604285+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:00:25.187 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'102 last_divergent_update: 28'102 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4619.0:1 2024-04-25T06:58:04.604285+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-04-25T06:58:04.613339+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: keeping 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-04-25T06:58:04.613339+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:00:25.188 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'103 last_divergent_update: 28'103 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-04-25T06:58:04.613339+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4616.0:1 2024-04-25T06:58:04.596127+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 20 _merge_object_divergent_entries: keeping 28'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4616.0:1 2024-04-25T06:58:04.596127+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'101 last_divergent_update: 28'101 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:00:25.189 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T06:59:52.639+0000 7fb4f7dc6640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:00:25.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:00:25.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:00:25.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:00:25.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:00:25.199 INFO:tasks.workunit.client.0.smithi088.stdout:success 2024-04-25T07:00:25.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-04-25T07:00:25.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.26017.out 2024-04-25T07:00:25.200 INFO:tasks.workunit.client.0.smithi088.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.26017.out': No such file or directory 2024-04-25T07:00:25.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-04-25T07:00:25.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:00:25.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:00:25.679 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T07:00:25.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-04-25T07:00:25.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:00:25.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:00:25.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:00:25.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:00:25.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:00:31.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:00:31.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:00:31.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:00:31.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:00:31.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:00:31.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:00:31.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:00:31.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:00:31.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:00:31.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:00:31.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:00:31.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:00:31.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:00:31.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:00:31.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:00:31.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:00:31.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:00:31.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:00:31.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:00:31.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:00:31.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:00:31.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:00:31.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:00:31.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:00:31.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:00:31.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:31.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:31.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T07:00:31.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:00:31.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:00:31.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:00:31.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:00:31.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:00:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:00:31.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:00:31.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:00:31.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:00:31.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:00:31.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:00:31.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:00:31.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:00:31.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:00:31.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:00:31.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:00:31.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:00:31.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:00:31.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:00:31.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:00:31.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:00:31.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:31.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:31.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T07:00:31.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:00:31.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:00:31.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:00:31.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:00:31.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:31.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:31.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26017 2024-04-25T07:00:31.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:00:31.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:00:31.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:00:31.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:00:31.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-04-25T07:00:31.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-04-25T07:00:31.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-04-25T07:00:31.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-04-25T07:00:31.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-04-25T07:00:31.076 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-04-25T07:00:31.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-04-25T07:00:31.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-04-25T07:00:31.078 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:00:31.078 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:00:31.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-04-25T07:00:31.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:00:31.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:00:31.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:00:31.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:00:31.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:00:31.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:00:31.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:00:31.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:00:31.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:00:31.450 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:00:31.450 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:31.450 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:31.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:00:31.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:00:31.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:00:31.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:00:31.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:00:31.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:00:31.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:00:31.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:00:31.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:00:31.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:00:31.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:00:31.652 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:00:31.652 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:31.652 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:31.653 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-mon.a.asok 2024-04-25T07:00:31.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:00:31.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26017/ceph-mon.a.asok config get fsid 2024-04-25T07:00:31.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:00:31.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:00:31.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:00:31.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:00:31.770 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:00:31.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:00:31.770 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:00:31.770 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:00:31.770 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:00:31.770 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:31.771 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:31.771 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-mon.a.asok 2024-04-25T07:00:31.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:00:31.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26017/ceph-mon.a.asok config get mon_host 2024-04-25T07:00:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-04-25T07:00:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:00:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:00:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:00:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:00:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:00:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:00:32.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:00:32.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:00:32.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:00:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:00:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:32.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:00:32.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:00:32.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:00:32.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:00:32.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-04-25T07:00:32.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:00:32.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:00:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:00:32.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:00:32.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:00:32.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:00:32.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8db453d6-6a31-457a-ac95-0d30560b9fc3 2024-04-25T07:00:32.876 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 8db453d6-6a31-457a-ac95-0d30560b9fc3 2024-04-25T07:00:32.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8db453d6-6a31-457a-ac95-0d30560b9fc3' 2024-04-25T07:00:32.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:00:32.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQ/ylmbD3RNRAALTKsVri6rg4grjGLyAXc2w== 2024-04-25T07:00:32.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQ/ylmbD3RNRAALTKsVri6rg4grjGLyAXc2w=="}' 2024-04-25T07:00:32.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8db453d6-6a31-457a-ac95-0d30560b9fc3 -i td/divergent-priors/0/new.json 2024-04-25T07:00:33.194 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:00:33.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:00:33.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQ/ylmbD3RNRAALTKsVri6rg4grjGLyAXc2w== --osd-uuid 8db453d6-6a31-457a-ac95-0d30560b9fc3 2024-04-25T07:00:33.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:33.258+0000 7f9b58d397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:33.262 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:33.262+0000 7f9b58d397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:33.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:33.262+0000 7f9b58d397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:33.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:33.262+0000 7f9b58d397c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:00:33.264 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:33.262+0000 7f9b58d397c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:00:33.265 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:33.262+0000 7f9b58d397c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:00:33.294 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:33.294+0000 7f9b58d397c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:00:39.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:00:39.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:00:39.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:00:39.139 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:00:39.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:00:39.735 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:00:39.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:00:39.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:00:39.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:00:39.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:00:39.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:00:39.775 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:39.774+0000 7fcbfa1df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:39.792 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:39.790+0000 7fcbfa1df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:39.794 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:39.794+0000 7fcbfa1df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:00:40.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:00:40.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:00:40.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:00:40.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:00:40.205 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:00:40.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:00:40.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:40.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:00:40.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:40.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:40.358 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:40.358+0000 7fcbfa1df7c0 -1 Falling back to public interface 2024-04-25T07:00:40.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:41.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:41.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:41.633 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:00:41.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:00:41.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:41.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:42.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:42.850 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:00:42.850+0000 7fcbfa1df7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:00:43.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:43.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:43.041 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:00:43.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:00:43.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:43.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:43.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:44.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:44.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:44.631 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:00:44.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:00:44.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:44.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:45.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:46.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:46.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:46.109 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:00:46.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:00:46.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:46.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:46.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:47.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:47.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:47.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:00:47.586 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:00:47.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:47.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:48.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:49.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:49.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:49.072 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:00:49.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:00:49.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:49.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:49.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:50.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:50.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:50.589 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:00:50.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:00:50.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:50.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:51.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:52.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:52.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:52.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:00:52.075 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:00:52.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:52.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:52.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:53.535 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:00:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:00:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:54.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:55.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:55.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:55.011 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:00:55.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:00:55.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:55.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:55.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:56.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:56.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:56.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:00:56.520 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:00:56.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:56.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:57.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:00:58.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:00:58.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:00:58.495 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:00:58.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:00:58.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:00:58.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:00:59.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:00.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:00.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:00.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:01:00.275 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:01:00.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:00.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:01:01.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:02.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:02.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:02.286 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:01:02.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:01:02.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:02.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:01:03.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:04.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:04.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:04.139 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:01:04.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:01:04.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:04.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:01:04.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:05.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:05.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:05.587 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:01:05.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:01:05.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:05.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:01:06.042 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3658298476,v1:127.0.0.1:6803/3658298476] [v2:127.0.0.1:6804/3658298476,v1:127.0.0.1:6805/3658298476] exists,up 8db453d6-6a31-457a-ac95-0d30560b9fc3 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:01:06.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:01:06.045 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:01:06.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:01:06.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:01:06.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:01:06.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:01:06.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:01:06.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:01:06.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:01:06.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:01:06.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2d99303-3733-4f44-8f02-eddc6e2cece0 2024-04-25T07:01:06.048 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 b2d99303-3733-4f44-8f02-eddc6e2cece0 2024-04-25T07:01:06.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b2d99303-3733-4f44-8f02-eddc6e2cece0' 2024-04-25T07:01:06.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:01:06.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCy/ylmoSS3BBAAl1hJX1sef+kXwReqidBYdA== 2024-04-25T07:01:06.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCy/ylmoSS3BBAAl1hJX1sef+kXwReqidBYdA=="}' 2024-04-25T07:01:06.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2d99303-3733-4f44-8f02-eddc6e2cece0 -i td/divergent-priors/1/new.json 2024-04-25T07:01:06.619 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:01:06.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:01:06.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCy/ylmoSS3BBAAl1hJX1sef+kXwReqidBYdA== --osd-uuid b2d99303-3733-4f44-8f02-eddc6e2cece0 2024-04-25T07:01:06.677 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:06.678+0000 7f76def2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:06.680 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:06.678+0000 7f76def2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:06.681 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:06.682+0000 7f76def2a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:06.681 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:06.682+0000 7f76def2a7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:01:06.681 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:06.682+0000 7f76def2a7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:01:06.681 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:06.682+0000 7f76def2a7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:01:06.724 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:06.722+0000 7f76def2a7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:01:12.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:01:12.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:01:12.704 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:01:12.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:01:12.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:01:13.292 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:01:13.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:01:13.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:01:13.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:01:13.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:01:13.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:01:13.326 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:13.326+0000 7f8baaca17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:13.338+0000 7f8baaca17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:13.338 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:13.338+0000 7f8baaca17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:13.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:01:13.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:01:13.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:01:13.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:01:13.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:01:13.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:13.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:01:13.712 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:01:13.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:13.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:13.886 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:13.886+0000 7f8baaca17c0 -1 Falling back to public interface 2024-04-25T07:01:14.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:15.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:15.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:15.150 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:01:15.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:01:15.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:15.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:15.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:16.480 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:16.478+0000 7f8baaca17c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:01:16.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:16.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:16.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:01:16.574 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:01:16.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:16.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:17.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:18.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:18.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:18.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:01:18.127 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:01:18.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:18.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:18.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:19.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:19.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:19.568 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:01:19.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:01:19.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:19.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:20.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:21.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:21.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:21.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:01:21.017 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:01:21.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:21.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:21.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:22.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:22.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:22.592 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:01:22.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:01:22.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:22.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:23.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:24.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:24.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:24.063 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:01:24.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:01:24.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:24.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:24.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:25.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:25.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:25.603 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:01:25.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:01:25.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:25.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:26.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:27.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:27.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:27.028 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:01:27.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:01:27.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:27.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:27.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:28.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:28.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:28.584 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:01:28.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:01:28.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:28.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:29.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:30.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:30.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:30.022 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:01:30.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:01:30.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:30.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:31.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:32.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:32.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:32.140 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:01:32.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:01:32.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:32.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:32.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:33.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:33.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:33.934 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:01:33.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:01:33.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:33.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:34.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:35.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:35.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:35.672 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:01:35.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:01:35.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:35.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:36.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:37.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:37.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:37.397 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:01:37.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:01:37.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:37.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:38.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:39.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:39.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:39.811 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:01:39.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:01:39.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:39.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:01:40.255 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1263838535,v1:127.0.0.1:6811/1263838535] [v2:127.0.0.1:6812/1263838535,v1:127.0.0.1:6813/1263838535] exists,up b2d99303-3733-4f44-8f02-eddc6e2cece0 2024-04-25T07:01:40.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:01:40.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:01:40.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:01:40.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:01:40.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:01:40.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:01:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:01:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:01:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:01:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:01:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:01:40.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:01:40.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6160fb33-0967-4995-9dc7-f5ae0db1f5a0 2024-04-25T07:01:40.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6160fb33-0967-4995-9dc7-f5ae0db1f5a0' 2024-04-25T07:01:40.262 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 6160fb33-0967-4995-9dc7-f5ae0db1f5a0 2024-04-25T07:01:40.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:01:40.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDU/ylmISGHERAAnLT2bfWx6jrcGQ9avgkV9w== 2024-04-25T07:01:40.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDU/ylmISGHERAAnLT2bfWx6jrcGQ9avgkV9w=="}' 2024-04-25T07:01:40.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6160fb33-0967-4995-9dc7-f5ae0db1f5a0 -i td/divergent-priors/2/new.json 2024-04-25T07:01:40.765 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:01:40.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:01:40.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDU/ylmISGHERAAnLT2bfWx6jrcGQ9avgkV9w== --osd-uuid 6160fb33-0967-4995-9dc7-f5ae0db1f5a0 2024-04-25T07:01:40.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:40.817+0000 7fbcfa4647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:40.817 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:40.821+0000 7fbcfa4647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:40.818 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:40.821+0000 7fbcfa4647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:40.819 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:40.821+0000 7fbcfa4647c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:01:40.819 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:40.821+0000 7fbcfa4647c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:01:40.819 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:40.821+0000 7fbcfa4647c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:01:40.863 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:40.865+0000 7fbcfa4647c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:01:46.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:01:46.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:01:46.961 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:01:46.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:01:46.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:01:47.554 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:01:47.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:01:47.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:01:47.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:01:47.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:01:47.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:01:47.596 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:47.597+0000 7f5858f087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:47.647 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:47.649+0000 7f5858f087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:47.649 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:47.653+0000 7f5858f087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:01:48.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:01:48.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:01:48.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:01:48.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:01:48.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:01:48.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:48.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:01:48.006 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:01:48.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:01:48.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:48.202 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:48.205+0000 7f5858f087c0 -1 Falling back to public interface 2024-04-25T07:01:48.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:49.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:49.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:49.438 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:01:49.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:01:49.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:49.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:01:49.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:50.657 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:01:50.661+0000 7f5858f087c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:01:50.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:50.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:50.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:01:50.913 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:01:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:01:51.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:52.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:52.404 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:01:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:01:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:01:52.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:53.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:53.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:53.969 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:01:53.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:01:53.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:53.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:01:54.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:55.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:55.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:55.558 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:01:55.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:01:55.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:55.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:01:56.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:57.017 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:01:57.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:01:57.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:57.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:01:57.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:58.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:58.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:58.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:01:58.523 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:01:58.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:58.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:01:58.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:01:59.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:01:59.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:01:59.963 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:01:59.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:01:59.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:01:59.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:00.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:02:01.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:02:01.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:02:01.621 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:02:01.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:02:01.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:02:01.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:02.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:02:03.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:02:03.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:02:03.199 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:02:03.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:02:03.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:02:03.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:04.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:02:05.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:02:05.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:02:05.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:02:05.183 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:02:05.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:02:05.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:06.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:02:07.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:02:07.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:02:07.014 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:02:07.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:02:07.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:02:07.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:07.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:02:08.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:02:08.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:02:08.718 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:02:08.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:02:08.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:02:08.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:09.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:02:10.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:02:10.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:02:10.262 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:02:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:02:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:02:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:10.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:02:11.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:02:11.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:02:11.874 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:02:11.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:02:11.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:02:11.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:12.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:02:13.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:02:13.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:02:13.657 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:02:13.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:02:13.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:02:13.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:02:14.100 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2730044711,v1:127.0.0.1:6819/2730044711] [v2:127.0.0.1:6820/2730044711,v1:127.0.0.1:6821/2730044711] exists,up 6160fb33-0967-4995-9dc7-f5ae0db1f5a0 2024-04-25T07:02:14.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:02:14.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:02:14.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:02:14.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-04-25T07:02:14.840 INFO:tasks.workunit.client.0.smithi088.stderr:noout is set 2024-04-25T07:02:14.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-04-25T07:02:15.551 INFO:tasks.workunit.client.0.smithi088.stderr:noin is set 2024-04-25T07:02:15.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-04-25T07:02:16.529 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T07:02:16.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-04-25T07:02:16.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:02:17.141 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T07:02:17.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:02:18.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-04-25T07:02:18.900 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 3 2024-04-25T07:02:18.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-04-25T07:02:19.578 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 2 2024-04-25T07:02:19.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-04-25T07:02:20.348 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 pg_autoscale_mode to on 2024-04-25T07:02:20.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-04-25T07:02:20.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-04-25T07:02:20.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1714028540 2024-04-25T07:02:20.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-04-25T07:02:20.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-04-25T07:02:20.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-04-25T07:02:20.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:02:20.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:02:20.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:02:20.800 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:02:20.800 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:02:20.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:02:20.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:20.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:02:20.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-04-25T07:02:20.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-04-25T07:02:20.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-04-25T07:02:20.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:20.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:02:21.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672976 2024-04-25T07:02:21.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672976 2024-04-25T07:02:21.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2024-04-25T07:02:21.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:21.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:02:21.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509446 2024-04-25T07:02:21.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509446 2024-04-25T07:02:21.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509446' 2024-04-25T07:02:21.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:21.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-04-25T07:02:21.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:21.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:02:21.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-04-25T07:02:21.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:21.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-04-25T07:02:21.305 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836502 2024-04-25T07:02:21.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-25T07:02:21.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:02:21.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836502 2024-04-25T07:02:21.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:21.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672976 2024-04-25T07:02:21.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:21.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:02:21.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672976 2024-04-25T07:02:21.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:21.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672976 2024-04-25T07:02:21.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-04-25T07:02:21.715 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672976 2024-04-25T07:02:21.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:02:22.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672976 2024-04-25T07:02:22.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:22.115 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509446 2024-04-25T07:02:22.115 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:02:22.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509446 2024-04-25T07:02:22.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:22.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509446 2024-04-25T07:02:22.119 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509446 2024-04-25T07:02:22.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-04-25T07:02:22.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:02:22.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509446 -lt 64424509446 2024-04-25T07:02:22.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-04-25T07:02:22.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:02:22.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:02:22.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:02:22.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:02:22.583 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:02:22.583 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:02:22.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:02:22.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:02:22.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:02:22.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:02:22.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:02:22.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:02:22.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:02:22.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:02:22.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:02:23.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:02:23.151 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:02:23.151 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:02:23.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:02:23.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:23.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:02:23.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-04-25T07:02:23.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-04-25T07:02:23.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-04-25T07:02:23.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:23.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:02:23.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T07:02:23.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T07:02:23.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978' 2024-04-25T07:02:23.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:23.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:02:23.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-04-25T07:02:23.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-04-25T07:02:23.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978 2-64424509448' 2024-04-25T07:02:23.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:23.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-04-25T07:02:23.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:23.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:02:23.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-04-25T07:02:23.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:23.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-04-25T07:02:23.643 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836504 2024-04-25T07:02:23.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-25T07:02:23.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:02:24.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836504 2024-04-25T07:02:24.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:24.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T07:02:24.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:24.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:02:24.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T07:02:24.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:24.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T07:02:24.054 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672978 2024-04-25T07:02:24.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T07:02:24.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:02:24.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672978 2024-04-25T07:02:24.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:02:25.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:02:25.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:02:25.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672978 2024-04-25T07:02:25.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:25.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-04-25T07:02:25.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:25.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:02:25.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-04-25T07:02:25.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:25.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-04-25T07:02:25.950 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509448 2024-04-25T07:02:25.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-25T07:02:25.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:02:26.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509448 2024-04-25T07:02:26.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:02:26.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:02:26.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:02:26.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:02:26.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:02:26.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:02:26.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:02:26.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:02:26.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:02:26.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:02:26.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:02:27.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:02:27.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:02:27.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:02:27.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:02:27.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:02:27.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:02:27.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:02:27.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-04-25T07:02:27.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-04-25T07:02:28.267 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:02:28.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-04-25T07:02:28.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-04-25T07:02:28.284 INFO:tasks.workunit.client.0.smithi088.stdout:primary and soon to be divergent is 1 2024-04-25T07:02:28.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:02:28.690 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:02:28.690 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:02:18.027579+0000 0'0 31:23 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:07:26.470829+0000 0 0 2024-04-25T07:02:28.690 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:02:28.690 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:02:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:02:28.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-04-25T07:02:28.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1714028548 2024-04-25T07:02:28.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1714028548 - 1714028540 2024-04-25T07:02:28.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=8 2024-04-25T07:02:28.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 8 -gt 300 ']' 2024-04-25T07:02:28.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-04-25T07:02:28.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-04-25T07:02:28.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-04-25T07:02:28.713 INFO:tasks.workunit.client.0.smithi088.stdout:writing initial objects 2024-04-25T07:02:28.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-04-25T07:02:28.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-04-25T07:02:28.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:28.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-04-25T07:02:28.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:28.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-04-25T07:02:29.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:29.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-04-25T07:02:29.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:29.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-04-25T07:02:29.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:29.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-04-25T07:02:29.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:29.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-04-25T07:02:29.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:29.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-04-25T07:02:29.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:29.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-04-25T07:02:29.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:29.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-04-25T07:02:30.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:30.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-04-25T07:02:30.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:30.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-04-25T07:02:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-04-25T07:02:30.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:30.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-04-25T07:02:30.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:30.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-04-25T07:02:30.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:30.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-04-25T07:02:31.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:31.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-04-25T07:02:31.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:31.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-04-25T07:02:31.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:31.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-04-25T07:02:31.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:31.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-04-25T07:02:31.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:31.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-04-25T07:02:31.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:31.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-04-25T07:02:32.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:32.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-04-25T07:02:32.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:32.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-04-25T07:02:32.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:32.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-04-25T07:02:32.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:32.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-04-25T07:02:32.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:32.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-04-25T07:02:32.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:32.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-04-25T07:02:32.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:32.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-04-25T07:02:33.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:33.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-04-25T07:02:33.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:33.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-04-25T07:02:33.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:33.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-04-25T07:02:33.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:33.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-04-25T07:02:33.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:33.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-04-25T07:02:33.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:33.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-04-25T07:02:34.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:34.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-04-25T07:02:34.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:34.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-04-25T07:02:34.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:34.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-04-25T07:02:34.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:34.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-04-25T07:02:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:34.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-04-25T07:02:34.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:34.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-04-25T07:02:35.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:35.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-04-25T07:02:35.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:35.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-04-25T07:02:35.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:35.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-04-25T07:02:35.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:35.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-04-25T07:02:35.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:35.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-04-25T07:02:35.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:35.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-04-25T07:02:35.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:35.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-04-25T07:02:36.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:36.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-04-25T07:02:36.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:36.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-04-25T07:02:36.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:36.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-04-25T07:02:36.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:36.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-04-25T07:02:36.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:36.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-04-25T07:02:37.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:37.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-04-25T07:02:37.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:37.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-04-25T07:02:37.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:37.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-04-25T07:02:37.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:37.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-04-25T07:02:37.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:37.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-04-25T07:02:38.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:38.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-04-25T07:02:38.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:38.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-04-25T07:02:38.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:38.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-04-25T07:02:38.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:38.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-04-25T07:02:38.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:38.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-04-25T07:02:38.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:38.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-04-25T07:02:39.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:39.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-04-25T07:02:39.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:39.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-04-25T07:02:39.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:39.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-04-25T07:02:39.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:39.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-04-25T07:02:39.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:39.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-04-25T07:02:40.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:40.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-04-25T07:02:40.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:40.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-04-25T07:02:41.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:41.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-04-25T07:02:41.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:41.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-04-25T07:02:41.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:41.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-04-25T07:02:41.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:41.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-04-25T07:02:41.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:41.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-04-25T07:02:42.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:42.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-04-25T07:02:42.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:42.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-04-25T07:02:42.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:42.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-04-25T07:02:42.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:42.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-04-25T07:02:42.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:42.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-04-25T07:02:42.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:42.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-04-25T07:02:43.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:43.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-04-25T07:02:43.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:43.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-04-25T07:02:43.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:43.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-04-25T07:02:43.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:43.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-04-25T07:02:43.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:43.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-04-25T07:02:43.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:43.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-04-25T07:02:44.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:44.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-04-25T07:02:44.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:44.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-04-25T07:02:44.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:44.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-04-25T07:02:44.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:44.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-04-25T07:02:44.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:44.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-04-25T07:02:44.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:44.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-04-25T07:02:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-04-25T07:02:45.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:45.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-04-25T07:02:45.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:45.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-04-25T07:02:45.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:45.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-04-25T07:02:46.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:46.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-04-25T07:02:46.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:46.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-04-25T07:02:47.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:02:47.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-04-25T07:02:47.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:02:47.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-04-25T07:02:47.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:02:47.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:02:47.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:02:47.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:02:47.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:02:47.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:02:47.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:02:47.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:02:47.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:02:47.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:02:47.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:02:47.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:02:47.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:02:47.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:02:47.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:02:48.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:02:48.187 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:02:48.187 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:02:48.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:02:48.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:48.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:02:48.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T07:02:48.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T07:02:48.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T07:02:48.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:48.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:02:48.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T07:02:48.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T07:02:48.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672985' 2024-04-25T07:02:48.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:02:48.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:02:48.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-04-25T07:02:48.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-04-25T07:02:48.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509455' 2024-04-25T07:02:48.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:48.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T07:02:48.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:48.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:02:48.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T07:02:48.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:48.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T07:02:48.706 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836511 2024-04-25T07:02:48.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T07:02:48.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:02:49.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836511 2024-04-25T07:02:49.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:02:50.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:02:50.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:02:50.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-04-25T07:02:50.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:50.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T07:02:50.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:50.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:02:50.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:50.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T07:02:50.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T07:02:50.681 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672985 2024-04-25T07:02:50.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T07:02:50.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:02:51.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-25T07:02:51.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:02:51.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:02:51.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-04-25T07:02:51.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:02:51.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-04-25T07:02:51.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:02:51.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-04-25T07:02:51.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-25T07:02:51.170 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509455 2024-04-25T07:02:51.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:02:51.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509455 2024-04-25T07:02:51.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:02:51.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:02:51.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:02:52.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:02:52.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:02:52.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:02:52.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:02:52.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:02:52.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:02:52.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:02:52.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:02:52.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-04-25T07:02:52.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:02:52.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:02:52.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:02:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-04-25T07:02:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:02:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:02:53.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-04-25T07:02:53.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:02:53.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:02:53.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:02:53.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:02:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-04-25T07:02:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-04-25T07:02:53.818 INFO:tasks.workunit.client.0.smithi088.stdout:blackholing osds 0 2 2024-04-25T07:02:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:02:54.360 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:02:54.360 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:48.019351+0000 34'68 38:128 [2,0,1] 2 [2,0,1] 2 34'68 2024-04-25T07:02:48.019206+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:05:57.741461+0000 3 0 2024-04-25T07:02:54.360 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:48.634725+0000 36'68 38:128 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:26:23.225996+0000 2 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 0 0 0 2525 0 0 15 54 15 active+clean 2024-04-25T07:02:46.756805+0000 34'69 38:121 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:49.028586+0000 34'68 38:128 [2,0,1] 2 [2,0,1] 2 34'68 2024-04-25T07:02:49.028388+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:10:04.624089+0000 3 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 3 0 0 0 0 1851 0 0 13 54 13 active+clean 2024-04-25T07:02:48.888162+0000 32'67 38:111 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:25:08.709572+0000 3 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:46.755713+0000 34'69 38:121 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:40.618027+0000 34'69 38:116 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 0 0 0 1292 0 0 13 54 13 active+clean 2024-04-25T07:02:46.752456+0000 32'67 38:108 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:48.635891+0000 34'69 38:116 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:46.757711+0000 34'69 38:121 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:02:42.576616+0000 32'67 38:122 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:55:06.535311+0000 2 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:02:46.533291+0000 36'68 38:109 [2,0,1] 2 [2,0,1] 2 32'67 2024-04-25T07:02:43.547323+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.361 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 0 0 0 617 0 0 14 54 14 active+clean 2024-04-25T07:02:48.635332+0000 34'68 38:123 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:57:51.546677+0000 0 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:02:40.622516+0000 34'68 38:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 5 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:46.758623+0000 34'69 38:120 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:45.603505+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:02:48.638962+0000 32'67 38:122 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:16:13.576405+0000 2 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:40.631784+0000 34'68 38:123 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:58:40.850103+0000 2 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:46.757092+0000 34'69 38:121 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:48.634839+0000 34'69 38:116 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 0 0 0 2468 0 0 13 54 13 active+clean 2024-04-25T07:02:46.980687+0000 32'67 38:116 [2,0,1] 2 [2,0,1] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:44:40.749001+0000 4 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:49.578976+0000 34'68 38:123 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:14:56.945153+0000 3 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 0 0 0 1292 0 0 13 54 13 active+clean 2024-04-25T07:02:46.533220+0000 32'67 38:108 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 0 0 0 617 0 0 13 54 13 active+clean 2024-04-25T07:02:49.759597+0000 32'67 38:111 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:55:30.252991+0000 1 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:46.752216+0000 34'69 38:121 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.362 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:40.623561+0000 34'69 38:116 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:02:46.753342+0000 34'68 38:119 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:02:48.636137+0000 34'68 38:115 [0,1,2] 0 [0,1,2] 0 34'68 2024-04-25T07:02:47.539949+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 0 0 0 1292 0 0 13 54 13 active+clean 2024-04-25T07:02:46.752380+0000 32'67 38:108 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:02:40.623571+0000 35'68 38:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 0 0 0 1292 0 0 13 54 13 active+clean 2024-04-25T07:02:46.758442+0000 32'67 38:108 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:02:48.636340+0000 34'68 38:115 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:02:46.754318+0000 36'68 38:109 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:02:54.363 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:02:54.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:02:54.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-04-25T07:02:54.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:02:54.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:02:54.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:02:54.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:02:54.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:02:54.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.0.asok 2024-04-25T07:02:54.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:02:54.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:02:54.482 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:02:54.482 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:02:54.482 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:02:54.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:02:54.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-04-25T07:02:54.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:02:54.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:02:54.495 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:02:54.495 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:02:54.495 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:02:54.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.2.asok 2024-04-25T07:02:54.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:02:54.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:02:54.605 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:02:54.606 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:02:54.606 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:02:54.618 INFO:tasks.workunit.client.0.smithi088.stdout:create new divergent objects 2024-04-25T07:02:54.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-04-25T07:02:54.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-04-25T07:02:54.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:02:54.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-04-25T07:02:54.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:02:54.621 INFO:tasks.workunit.client.0.smithi088.stdout:writing divergent objects 2024-04-25T07:02:54.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-04-25T07:02:54.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-04-25T07:02:54.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-04-25T07:02:54.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:02:54.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:02:54.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-04-25T07:02:54.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-04-25T07:02:54.624 INFO:tasks.workunit.client.0.smithi088.stdout:remove divergent objects 2024-04-25T07:02:54.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-04-25T07:02:54.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-04-25T07:02:54.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:02:54.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-04-25T07:02:54.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-04-25T07:02:54.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:02:54.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-04-25T07:02:54.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-04-25T07:02:54.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-04-25T07:02:54.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-04-25T07:02:54.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-04-25T07:03:04.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-04-25T07:03:04.638 INFO:tasks.workunit.client.0.smithi088.stderr:rados: no process found 2024-04-25T07:03:04.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-04-25T07:03:04.639 INFO:tasks.workunit.client.0.smithi088.stdout:killing all the osds 2024-04-25T07:03:04.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:53.781835+0000 34'68 45:137 [1,2,0] 1 [1,2,0] 1 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:36:46.538294+0000 3 0 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:58.025806+0000 36'68 45:131 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:37.353539+0000 2 0 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 0 0 0 2468 0 0 15 54 15 active+clean 2024-04-25T07:02:55.069677+0000 34'69 44:145 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:55.069431+0000 34'69 2024-04-25T07:02:55.069431+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:14:20.508290+0000 4 0 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:57.636705+0000 34'68 45:137 [1,2,0] 1 [1,2,0] 1 34'68 2024-04-25T07:02:57.636504+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:51:58.644824+0000 3 0 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 3 0 0 0 0 1851 0 0 13 54 13 active+clean 2024-04-25T07:02:48.888162+0000 32'67 45:130 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:26:06.313497+0000 3 0 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 5 0 0 0 0 3435 0 0 16 54 16 active+clean 2024-04-25T07:02:57.027050+0000 41'70 44:146 [2,0,1] 2 [2,0,1] 2 41'70 2024-04-25T07:02:57.026851+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:00:31.757595+0000 5 0 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:02:51.532530+0000 34'69 45:143 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:32:39.140824+0000 3 0 2024-04-25T07:03:05.041 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:02:59.547675+0000 32'67 45:136 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:59.547469+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:22.604465+0000 2 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:58.118342+0000 34'69 45:124 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:02:51.774330+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 0 0 0 3702 0 0 15 54 15 active+clean 2024-04-25T07:02:53.093533+0000 34'69 44:145 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:51:30.231154+0000 6 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:02:42.576616+0000 32'67 45:141 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:12:56.533411+0000 2 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 5 0 0 0 0 3085 0 0 14 54 14 active+clean 2024-04-25T07:02:58.637639+0000 36'68 45:134 [1,2,0] 1 [1,2,0] 1 36'68 2024-04-25T07:02:58.637424+0000 36'68 2024-04-25T07:02:58.637424+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:10:29.491523+0000 5 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 0 0 0 617 0 0 14 54 14 active+clean 2024-04-25T07:02:48.635332+0000 34'68 45:142 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:03:34.978676+0000 0 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:55.659474+0000 34'68 45:142 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:55.659162+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:19:30.696236+0000 3 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 0 0 0 1234 0 0 15 54 15 active+clean 2024-04-25T07:02:54.006770+0000 34'69 44:144 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:27.214776+0000 2 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:02:48.638962+0000 32'67 45:141 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:25:43.719926+0000 2 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:40.631784+0000 34'68 45:142 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:52:41.738096+0000 2 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:02:51.065375+0000 34'69 44:145 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:49:32.143741+0000 3 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 0 0 0 1851 0 0 17 54 17 active+clean 2024-04-25T07:02:58.118150+0000 41'71 45:145 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:56:21.782068+0000 4 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 0 0 0 2468 0 0 13 54 13 active+clean 2024-04-25T07:02:52.730686+0000 32'67 45:136 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:58:42.023750+0000 4 0 2024-04-25T07:03:05.042 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:49.578976+0000 34'68 45:142 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:58:48.831053+0000 3 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 0 0 0 1234 0 0 14 54 14 active+clean 2024-04-25T07:02:55.992163+0000 41'68 44:133 [2,0,1] 2 [2,0,1] 2 41'68 2024-04-25T07:02:55.991960+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:52:10.703171+0000 2 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 0 0 0 617 0 0 13 54 13 active+clean 2024-04-25T07:02:49.759597+0000 32'67 45:130 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:33:50.352724+0000 1 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:02:52.858085+0000 34'69 45:130 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:02:52.731533+0000 34'69 45:143 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:01:52.959166+0000 3 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:50.078306+0000 34'68 44:143 [2,0,1] 2 [2,0,1] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:11:38.703787+0000 3 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 0 0 0 2468 0 0 14 54 14 active+clean 2024-04-25T07:02:58.118340+0000 34'68 45:131 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:42:58.981787+0000 4 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 1 0 0 0 0 617 0 0 13 54 13 active+clean 2024-04-25T07:02:56.559128+0000 32'67 45:136 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:56.558814+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:45:36.349246+0000 1 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:02:50.576623+0000 35'68 45:142 [1,0,2] 1 [1,0,2] 1 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:42:53.513741+0000 3 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:02:52.125413+0000 32'67 44:132 [2,0,1] 2 [2,0,1] 2 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:16:30.650005+0000 2 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 4 0 0 0 0 2468 0 0 14 54 14 active+clean 2024-04-25T07:02:52.859700+0000 34'68 45:142 [0,1,2] 0 [0,1,2] 0 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:26:31.265899+0000 4 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 6 0 0 0 0 3435 0 0 16 54 16 active+clean 2024-04-25T07:02:54.593671+0000 41'70 45:135 [1,2,0] 1 [1,2,0] 1 36'68 2024-04-25T07:02:54.593475+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:59:01.997190+0000 5 0 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:03:05.043 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:03:05.044 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:03:05.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:03:05.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:03:05.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:03:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:03:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:03:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:03:05.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:03:05.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:03:05.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-04-25T07:03:05.880 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T07:03:05.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:03:05.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-04-25T07:03:06.425 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T07:03:06.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:03:06.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-04-25T07:03:06.974 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already down. 2024-04-25T07:03:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:03:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-04-25T07:03:07.873 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T07:03:07.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:03:07.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-04-25T07:03:08.436 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already out. 2024-04-25T07:03:08.453 INFO:tasks.workunit.client.0.smithi088.stdout:bringing up non_divergent 0 2 2024-04-25T07:03:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-04-25T07:03:08.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:03:08.868 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:03:08.868 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 0 0 0 1851 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:53.781835+0000 34'68 49:147 [1,2,0] 1 [1,2,0] 1 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:38:40.958072+0000 3 0 2024-04-25T07:03:08.868 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 0 0 0 1851 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:58.025806+0000 36'68 49:141 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:37:10.688249+0000 2 0 2024-04-25T07:03:08.868 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 0 0 0 2468 0 0 15 54 15 stale+active+clean 2024-04-25T07:02:55.069677+0000 34'69 49:158 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:55.069431+0000 34'69 2024-04-25T07:02:55.069431+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:33.144834+0000 4 0 2024-04-25T07:03:08.868 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 0 0 0 1851 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:57.636705+0000 34'68 49:147 [1,2,0] 1 [1,2,0] 1 34'68 2024-04-25T07:02:57.636504+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:53:50.518656+0000 3 0 2024-04-25T07:03:08.868 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 3 0 0 0 0 1851 0 0 13 54 13 stale+active+clean 2024-04-25T07:02:48.888162+0000 32'67 45:130 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:26:06.313497+0000 3 0 2024-04-25T07:03:08.868 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 5 0 0 0 0 3435 0 0 16 54 16 stale+active+clean 2024-04-25T07:03:04.586061+0000 41'70 49:147 [1,2,0] 1 [1,2,0] 1 41'70 2024-04-25T07:03:04.585868+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:44:58.748592+0000 5 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 3 0 0 0 0 1851 0 0 15 54 15 stale+active+clean 2024-04-25T07:03:02.043558+0000 34'69 49:158 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:02:56.767904+0000 3 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 0 0 0 1234 0 0 13 54 13 stale+active+clean 2024-04-25T07:02:59.547675+0000 32'67 49:146 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:59.547469+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:38:33.255350+0000 2 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 0 0 0 2468 0 0 15 54 15 stale+active+clean 2024-04-25T07:03:01.636683+0000 34'69 49:142 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:03:01.636353+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:24:28.511838+0000 4 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 0 0 0 3702 0 0 15 54 15 stale+active+clean 2024-04-25T07:02:53.093533+0000 34'69 49:158 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:55:30.061721+0000 6 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 0 0 0 1234 0 0 13 54 13 stale+active+clean 2024-04-25T07:02:42.576616+0000 32'67 49:151 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:05:37.966546+0000 2 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 5 0 0 0 0 3085 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:58.637639+0000 36'68 49:144 [1,2,0] 1 [1,2,0] 1 36'68 2024-04-25T07:02:58.637424+0000 36'68 2024-04-25T07:02:58.637424+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:22:38.868793+0000 5 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 0 0 0 617 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:48.635332+0000 34'68 45:142 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:03:34.978676+0000 0 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 0 0 0 1851 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:55.659474+0000 34'68 49:152 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:55.659162+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:55:52.368515+0000 3 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 0 0 0 1234 0 0 15 54 15 stale+active+clean 2024-04-25T07:02:54.006770+0000 34'69 49:157 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:31:35.431231+0000 2 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 0 0 0 1234 0 0 13 54 13 stale+active+clean 2024-04-25T07:02:48.638962+0000 32'67 49:151 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:46:10.666933+0000 2 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 0 0 0 1851 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:40.631784+0000 34'68 49:152 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:04:45.648602+0000 2 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 0 0 0 1851 0 0 15 54 15 stale+active+clean 2024-04-25T07:02:51.065375+0000 34'69 49:158 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:08:41.643659+0000 3 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 0 0 0 1851 0 0 17 54 17 stale+active+clean 2024-04-25T07:02:58.118150+0000 41'71 49:155 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:24:39.098585+0000 4 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 0 0 0 2468 0 0 13 54 13 stale+active+clean 2024-04-25T07:02:52.730686+0000 32'67 49:146 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:44:54.306473+0000 4 0 2024-04-25T07:03:08.869 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 0 0 0 1851 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:49.578976+0000 34'68 49:152 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:06:59.220686+0000 3 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 0 0 0 1234 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:55.992163+0000 41'68 49:146 [2,0,1] 2 [2,0,1] 2 41'68 2024-04-25T07:02:55.991960+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:58:14.134900+0000 2 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 0 0 0 617 0 0 13 54 13 stale+active+clean 2024-04-25T07:02:49.759597+0000 32'67 45:130 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:33:50.352724+0000 1 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 3 0 0 0 0 1851 0 0 15 54 15 stale+active+clean 2024-04-25T07:03:00.588656+0000 34'69 49:148 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:03:00.588452+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:35:21.704613+0000 3 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 3 0 0 0 0 1851 0 0 15 54 15 stale+active+clean 2024-04-25T07:02:52.731533+0000 34'69 49:153 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:13:01.265950+0000 3 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 0 0 0 1851 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:50.078306+0000 34'68 49:156 [2,0,1] 2 [2,0,1] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:36:43.830643+0000 3 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 0 0 0 2468 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:58.118340+0000 34'68 49:141 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:18:40.456606+0000 4 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 1 0 0 0 0 617 0 0 13 54 13 stale+active+clean 2024-04-25T07:02:56.559128+0000 32'67 49:146 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:56.558814+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:21:19.595050+0000 1 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 0 0 0 1851 0 0 14 54 14 stale+active+clean 2024-04-25T07:03:02.043456+0000 35'68 49:157 [1,2,0] 1 [1,2,0] 1 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:21:42.072243+0000 3 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 0 0 0 1292 0 0 13 54 13 stale+active+clean 2024-04-25T07:03:03.974638+0000 32'67 49:114 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 4 0 0 0 0 2468 0 0 14 54 14 stale+active+clean 2024-04-25T07:02:52.859700+0000 34'68 45:142 [0,1,2] 0 [0,1,2] 0 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:26:31.265899+0000 4 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 6 0 0 0 0 3435 0 0 16 54 16 stale+active+clean 2024-04-25T07:02:54.593671+0000 41'70 49:145 [1,2,0] 1 [1,2,0] 1 36'68 2024-04-25T07:02:54.593475+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:08:29.378318+0000 5 0 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:03:08.870 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:03:08.871 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:03:08.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:03:08.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:03:08.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:03:08.888 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:03:08.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:03:08.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:03:08.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:03:08.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:03:08.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:03:08.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:03:08.920 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:08.920+0000 7f53bebdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:08.922 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:08.924+0000 7f53bebdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:08.923 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:08.924+0000 7f53bebdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:09.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:03:09.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:03:09.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:03:09.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:03:09.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:03:09.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:09.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:03:09.401 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:03:09.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:09.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:09.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:09.488+0000 7f53bebdf7c0 -1 Falling back to public interface 2024-04-25T07:03:09.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:03:10.818 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:03:10.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:10.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:11.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:12.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:12.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:12.282 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:03:12.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:03:12.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:12.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:12.738 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:12.740+0000 7f53bebdf7c0 -1 osd.0 41 log_to_monitors true 2024-04-25T07:03:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:13.713 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:03:13.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:03:13.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:13.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:03:14.215 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up out weight 0 up_from 60 up_thru 37 down_at 51 last_clean_interval [5,41) [v2:127.0.0.1:6802/106036612,v1:127.0.0.1:6803/106036612] [v2:127.0.0.1:6804/106036612,v1:127.0.0.1:6805/106036612] exists,up 8db453d6-6a31-457a-ac95-0d30560b9fc3 2024-04-25T07:03:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:03:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:03:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:03:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:03:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-04-25T07:03:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:03:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:03:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:03:14.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:03:14.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:03:14.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:03:14.221 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:03:14.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:03:14.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:03:14.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:03:14.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:03:14.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:03:14.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:03:14.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:03:14.259 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:14.260+0000 7f490de587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:14.284 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:14.284+0000 7f490de587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:14.285 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:14.288+0000 7f490de587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:03:14.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:03:14.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:03:14.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:03:14.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:03:14.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:03:14.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:14.660 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:03:14.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:03:14.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:14.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:03:14.846 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:14.848+0000 7f490de587c0 -1 Falling back to public interface 2024-04-25T07:03:15.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:16.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:16.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:16.080 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:03:16.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:03:16.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:16.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:03:16.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:17.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:17.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:17.537 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:03:17.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:03:17.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:17.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:03:18.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:03:18.121 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:03:18.124+0000 7f490de587c0 -1 osd.2 41 log_to_monitors true 2024-04-25T07:03:19.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:03:19.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:03:19.057 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:03:19.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:03:19.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:03:19.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:03:19.483 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up out weight 0 up_from 65 up_thru 35 down_at 53 last_clean_interval [15,41) [v2:127.0.0.1:6810/2515045402,v1:127.0.0.1:6811/2515045402] [v2:127.0.0.1:6812/2515045402,v1:127.0.0.1:6813/2515045402] exists,up 6160fb33-0967-4995-9dc7-f5ae0db1f5a0 2024-04-25T07:03:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:03:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:03:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:03:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:03:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-04-25T07:03:19.999 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already in. 2024-04-25T07:03:20.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:03:20.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-04-25T07:03:20.817 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already in. 2024-04-25T07:03:20.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:03:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-04-25T07:03:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:03:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:03:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:03:20.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:03:20.842 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:03:20.842 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:03:20.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:03:20.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:03:20.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:03:20.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:03:20.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:03:20.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:03:20.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:03:20.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:03:20.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:03:21.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:03:21.596 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:03:21.596 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:03:21.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:03:21.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:03:21.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:03:21.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=257698037764 2024-04-25T07:03:21.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 257698037764 2024-04-25T07:03:21.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-257698037764' 2024-04-25T07:03:21.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:03:21.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:03:22.101 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:03:22.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:03:22.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:03:22.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:03:22.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:03:22.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:03:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=279172874243 2024-04-25T07:03:22.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 279172874243 2024-04-25T07:03:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-257698037764 2-279172874243' 2024-04-25T07:03:22.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:03:22.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-257698037764 2024-04-25T07:03:22.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:03:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:03:22.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-257698037764 2024-04-25T07:03:22.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:03:22.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=257698037764 2024-04-25T07:03:22.334 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 257698037764 2024-04-25T07:03:22.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 257698037764' 2024-04-25T07:03:22.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:03:22.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 257698037762 -lt 257698037764 2024-04-25T07:03:22.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:03:23.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:03:23.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:03:24.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 257698037764 -lt 257698037764 2024-04-25T07:03:24.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:03:24.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-279172874243 2024-04-25T07:03:24.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:03:24.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:03:24.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-279172874243 2024-04-25T07:03:24.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:03:24.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=279172874243 2024-04-25T07:03:24.248 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 279172874243 2024-04-25T07:03:24.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 279172874243' 2024-04-25T07:03:24.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:03:24.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 279172874243 -lt 279172874243 2024-04-25T07:03:24.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:03:24.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:24.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:25.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:03:25.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:25.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:25.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:25.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:25.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:25.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:25.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:25.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:25.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:25.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:25.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:03:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:03:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:03:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:03:26.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:03:26.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:03:26.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:26.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:26.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:26.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:26.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:26.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:26.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:26.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:26.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:26.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:26.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:27.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:27.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:03:27.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:03:27.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:03:27.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:03:27.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:03:27.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:03:27.243 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:03:27.243 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:03:27.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:03:27.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:03:27.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:03:27.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:03:27.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:03:27.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:03:27.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:27.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:27.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:27.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:27.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:27.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:27.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:28.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:28.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:28.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:28.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:28.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:28.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:03:28.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:03:28.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:03:28.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:03:28.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:03:28.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:03:28.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:03:28.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:03:29.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:03:29.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:03:29.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:03:29.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:03:29.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:03:29.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:03:29.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:29.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:29.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:29.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:29.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:29.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:29.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:30.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:30.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:30.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:30.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:30.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:30.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:03:30.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:03:30.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:03:30.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:03:30.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:03:30.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:03:30.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:03:30.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:03:31.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:03:31.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:03:31.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:03:31.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:03:31.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:03:32.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:03:32.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:32.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:32.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:32.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:32.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:32.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:32.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:32.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:32.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:32.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:32.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:33.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:33.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:03:33.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:03:33.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:03:33.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:03:33.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:03:33.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:03:33.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:03:33.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:03:33.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:03:33.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:03:33.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:03:33.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:03:33.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:03:35.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:03:35.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:35.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:35.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:35.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:35.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:35.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:35.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:35.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:35.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:35.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:35.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:36.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:36.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:03:36.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:03:36.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:03:36.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:03:36.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:03:36.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:03:36.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:03:36.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:03:36.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:03:36.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:03:36.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:03:36.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:03:36.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:03:40.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:03:40.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:40.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:40.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:40.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:40.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:40.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:40.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:40.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:40.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:40.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:40.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:41.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:41.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:03:41.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:03:41.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:03:41.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:03:41.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:03:41.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:03:41.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:03:41.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:03:41.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:03:41.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:03:41.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:03:41.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:03:41.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:03:48.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:03:48.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:48.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:48.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:48.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:48.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:48.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:48.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:48.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:48.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:48.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:48.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:03:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:03:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:03:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:03:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:03:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:03:48.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:03:48.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:03:49.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:03:49.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:03:49.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:03:49.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:03:49.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:03:56.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:03:56.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:03:56.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:03:56.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:03:56.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:03:56.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:03:56.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:03:56.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:03:57.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:03:57.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:03:57.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:03:57.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:03:57.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:03:57.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:03:57.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:03:57.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:03:57.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:03:57.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:03:57.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:03:57.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:03:57.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:03:58.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:03:58.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:03:58.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:03:58.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "3dc17557-779b-4d76-bc92-991d56bbc098", 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-04-25T07:03:58.897730+0000", 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-04-25T07:03:58.895 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 17 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "OSD_DOWN": { 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 osds down", 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:03:58.896 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "osd.1 (root=default,host=smithi088) is down" 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 97/291 objects degraded (33.333%), 32 pgs degraded", 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 32 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.897 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.898 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.899 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.900 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.901 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.902 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.903 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.904 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-04-25T07:03:58.905 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:00:31.110671Z", 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T07:00:31.110671Z", 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-04-25T07:03:58.906 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-04-25T07:03:58.907 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-04-25T07:03:58.908 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 73, 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T07:00:31.992003+0000", 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:03:21.948822+0000", 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-04-25T07:03:18.463446+0000", 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-04-25T07:03:20.377028+0000", 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:03:58.909 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5800384, 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "nodown", 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "noin", 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "noout", 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 9, 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:03:58.910 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 3, 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "test", 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T07:02:16.874982+0000", 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 1, 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool", 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T07:03:58.911 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "on", 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 32, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 19, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 32, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 32, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 32, 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T07:03:58.912 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "72", 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T07:03:58.913 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:03:58.914 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "33", 2024-04-25T07:03:58.914 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T07:03:58.914 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:03:58.914 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 0, 2024-04-25T07:03:58.914 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 0, 2024-04-25T07:03:58.914 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-04-25T07:03:58.915 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T07:03:58.915 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T07:03:58.915 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T07:03:58.915 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T07:03:58.915 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T07:03:58.915 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T07:03:58.915 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T07:03:58.916 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T07:03:58.916 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T07:03:58.916 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T07:03:58.916 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:03:58.916 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:03:58.916 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:03:58.916 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T07:03:58.917 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T07:03:58.917 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-04-25T07:03:58.917 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T07:03:58.917 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T07:03:58.917 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.917 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T07:03:58.917 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T07:03:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T07:03:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:03:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:03:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:03:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:03:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T07:03:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-04-25T07:03:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T07:03:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T07:03:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T07:03:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {}, 2024-04-25T07:03:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-04-25T07:03:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-04-25T07:03:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 1.7799999713897705, 2024-04-25T07:03:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 1.7799999713897705, 2024-04-25T07:03:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:03:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 1.190000057220459, 2024-04-25T07:03:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 1.190000057220459, 2024-04-25T07:03:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:03:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-04-25T07:03:58.921 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:03:58.921 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.921 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.921 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.921 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-04-25T07:03:58.921 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.922 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:03:58.922 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "8db453d6-6a31-457a-ac95-0d30560b9fc3", 2024-04-25T07:03:58.922 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T07:03:58.922 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:03:58.922 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:03:58.922 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:03:58.922 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 5, 2024-04-25T07:03:58.923 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 41, 2024-04-25T07:03:58.923 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 60, 2024-04-25T07:03:58.923 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 69, 2024-04-25T07:03:58.923 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 51, 2024-04-25T07:03:58.923 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:03:58.923 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:03:58.923 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.924 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.924 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.924 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:03:58.924 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:03:58.924 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.924 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.924 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.925 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:03:58.925 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:03:58.925 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.925 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.925 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.925 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:03:58.925 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.926 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.926 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.926 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:03:58.926 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:03:58.926 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.926 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.927 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.927 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:03:58.927 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:03:58.927 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.927 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.927 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.927 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:03:58.928 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.928 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.928 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.928 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:03:58.928 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:03:58.928 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.929 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.929 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.929 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:03:58.929 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:03:58.929 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.929 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.929 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.930 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:03:58.930 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.930 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.930 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.930 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:03:58.931 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:03:58.931 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.931 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.931 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.931 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:03:58.931 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:03:58.931 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.932 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.932 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.932 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6803/106036612", 2024-04-25T07:03:58.932 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6805/106036612", 2024-04-25T07:03:58.932 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6809/106036612", 2024-04-25T07:03:58.932 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6807/106036612", 2024-04-25T07:03:58.932 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:03:58.932 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T07:03:58.933 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T07:03:58.933 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.933 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.933 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.933 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:03:58.933 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "b2d99303-3733-4f44-8f02-eddc6e2cece0", 2024-04-25T07:03:58.933 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 0, 2024-04-25T07:03:58.934 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:03:58.934 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:03:58.934 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:03:58.934 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T07:03:58.934 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T07:03:58.934 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 10, 2024-04-25T07:03:58.935 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 48, 2024-04-25T07:03:58.935 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 52, 2024-04-25T07:03:58.935 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:03:58.935 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:03:58.935 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.935 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.935 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.936 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:03:58.936 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:03:58.936 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.936 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.936 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.936 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:03:58.936 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:03:58.937 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.937 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.937 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.937 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:03:58.937 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.937 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.938 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.938 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:03:58.938 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:03:58.938 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.938 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.939 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.939 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:03:58.939 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:03:58.939 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.939 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.939 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.939 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:03:58.940 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.940 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.940 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.940 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:03:58.940 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:03:58.940 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.940 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.940 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.941 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:03:58.941 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:03:58.941 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.941 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.941 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.941 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:03:58.941 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.941 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.942 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.942 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:03:58.942 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:03:58.942 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.942 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.942 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.942 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:03:58.943 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:03:58.943 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.943 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.943 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.943 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/1263838535", 2024-04-25T07:03:58.943 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/1263838535", 2024-04-25T07:03:58.943 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1263838535", 2024-04-25T07:03:58.943 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1263838535", 2024-04-25T07:03:58.944 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:03:58.944 INFO:tasks.workunit.client.0.smithi088.stdout: "exists" 2024-04-25T07:03:58.944 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.944 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.944 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.944 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:03:58.944 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "6160fb33-0967-4995-9dc7-f5ae0db1f5a0", 2024-04-25T07:03:58.944 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T07:03:58.945 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:03:58.945 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:03:58.945 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:03:58.945 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 15, 2024-04-25T07:03:58.945 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 41, 2024-04-25T07:03:58.945 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 65, 2024-04-25T07:03:58.945 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 71, 2024-04-25T07:03:58.945 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 53, 2024-04-25T07:03:58.946 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:03:58.946 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:03:58.946 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.946 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.946 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.946 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:03:58.946 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.947 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:03:58.948 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.948 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.948 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.948 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:03:58.948 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:03:58.948 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.948 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.948 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.949 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:03:58.949 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:03:58.949 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.949 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.949 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.949 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:03:58.949 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.949 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.950 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.950 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:03:58.950 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:03:58.950 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.950 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.950 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.950 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:03:58.951 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:03:58.951 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.951 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.951 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.951 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:03:58.951 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:03:58.951 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.951 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:03:58.952 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:03:58.952 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:03:58.952 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.952 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.952 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:03:58.952 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:03:58.953 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:03:58.953 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.953 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.953 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.953 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/2515045402", 2024-04-25T07:03:58.953 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/2515045402", 2024-04-25T07:03:58.954 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2515045402", 2024-04-25T07:03:58.954 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2515045402", 2024-04-25T07:03:58.954 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:03:58.954 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T07:03:58.954 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T07:03:58.954 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.955 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.955 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.955 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-04-25T07:03:58.955 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.955 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:03:58.955 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:03:05.648204+0000", 2024-04-25T07:03:58.955 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:03:58.956 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:03:58.956 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:03:58.956 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:03:58.956 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:01:04.261621+0000", 2024-04-25T07:03:58.956 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:03:58.956 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.957 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.957 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:03:58.957 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:03:06.267405+0000", 2024-04-25T07:03:58.957 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:03:58.957 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:03:58.957 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:03:58.958 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:03:58.958 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:01:38.903948+0000", 2024-04-25T07:03:58.958 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:03:58.958 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.958 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.958 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:03:58.959 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:03:06.809043+0000", 2024-04-25T07:03:58.959 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:03:58.959 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:03:58.959 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:03:58.959 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:03:58.959 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:02:13.056900+0000", 2024-04-25T07:03:58.960 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:03:58.960 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.960 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.960 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-04-25T07:03:58.960 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-04-25T07:03:58.960 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-04-25T07:03:58.961 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-04-25T07:03:58.961 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-04-25T07:03:58.961 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-04-25T07:03:58.961 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-04-25T07:03:58.961 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-04-25T07:03:58.961 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-04-25T07:03:58.961 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T07:03:58.962 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-04-25T07:03:58.962 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T07:03:58.962 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-04-25T07:03:58.962 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.962 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.962 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [], 2024-04-25T07:03:58.963 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [], 2024-04-25T07:03:58.963 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-04-25T07:03:58.963 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-04-25T07:03:58.963 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-04-25T07:03:58.963 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-04-25T07:03:58.963 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-04-25T07:03:58.964 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-04-25T07:03:58.964 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:03:58.964 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:03:58.964 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-04-25T07:03:58.964 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.964 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.964 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-04-25T07:03:58.965 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.965 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:03:58.965 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:03:58.965 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6804/106036612,v1:127.0.0.1:6805/106036612]", 2024-04-25T07:03:58.965 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:03:58.965 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:03:58.965 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:03:58.965 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:03:58.966 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:03:58.966 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:03:58.966 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:03:58.966 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:03:58.966 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:03:58.966 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:03:58.967 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:03:58.967 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:03:58.967 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:03:58.967 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:03:58.967 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:03:58.967 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:03:58.967 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:03:58.967 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:03:58.968 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:03:58.968 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:03:58.968 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:03:58.968 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:03:58.968 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:00:38.461379Z", 2024-04-25T07:03:58.968 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:03:58.968 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:03:58.968 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:03:58.969 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:03:58.969 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:03:58.969 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:03:58.969 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:03:58.969 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6802/106036612,v1:127.0.0.1:6803/106036612]", 2024-04-25T07:03:58.970 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:03:58.970 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/106036612,v1:127.0.0.1:6809/106036612]", 2024-04-25T07:03:58.970 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/106036612,v1:127.0.0.1:6807/106036612]", 2024-04-25T07:03:58.970 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:03:58.970 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:03:58.970 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:03:58.971 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:03:58.971 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:03:58.971 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:03:58.971 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:03:58.971 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:03:58.971 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:03:58.971 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:03:58.972 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:03:58.972 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:03:58.972 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:03:58.972 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.972 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.973 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:03:58.973 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:03:58.973 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/1263838535,v1:127.0.0.1:6813/1263838535]", 2024-04-25T07:03:58.973 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:03:58.973 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:03:58.973 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:03:58.973 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:03:58.974 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:03:58.974 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:03:58.974 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:03:58.974 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:03:58.974 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:03:58.974 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:03:58.975 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:03:58.975 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:03:58.975 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:03:58.975 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:03:58.975 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:03:58.975 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:03:58.976 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:03:58.976 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:03:58.976 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:03:58.976 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:03:58.976 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:03:58.976 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:03:58.977 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:01:11.931407Z", 2024-04-25T07:03:58.977 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:03:58.977 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:03:58.977 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:03:58.977 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:03:58.977 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:03:58.977 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:03:58.977 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:03:58.978 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/1263838535,v1:127.0.0.1:6811/1263838535]", 2024-04-25T07:03:58.978 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:03:58.978 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1263838535,v1:127.0.0.1:6817/1263838535]", 2024-04-25T07:03:58.978 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1263838535,v1:127.0.0.1:6815/1263838535]", 2024-04-25T07:03:58.978 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:03:58.978 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:03:58.978 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:03:58.979 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:03:58.979 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:03:58.979 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:03:58.979 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:03:58.979 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:03:58.979 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:03:58.979 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:03:58.980 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:03:58.980 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:03:58.980 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:03:58.980 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.980 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.980 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:03:58.980 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:03:58.980 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/2515045402,v1:127.0.0.1:6813/2515045402]", 2024-04-25T07:03:58.981 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:03:58.981 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:03:58.981 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:03:58.981 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:03:58.981 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:03:58.981 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:03:58.981 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:03:58.981 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:03:58.982 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:03:58.982 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:03:58.982 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:03:58.982 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:03:58.982 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:03:58.982 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:03:58.982 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:03:58.983 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:03:58.983 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:03:58.983 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:03:58.983 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:03:58.983 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:03:58.983 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:03:58.984 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:03:58.984 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:01:46.288989Z", 2024-04-25T07:03:58.984 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:03:58.984 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:03:58.984 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:03:58.984 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:03:58.985 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:03:58.985 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:03:58.985 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:03:58.985 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/2515045402,v1:127.0.0.1:6811/2515045402]", 2024-04-25T07:03:58.985 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2515045402,v1:127.0.0.1:6817/2515045402]", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2515045402,v1:127.0.0.1:6815/2515045402]", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:03:58.986 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [] 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 61 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.987 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 65 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 73, 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:03:58.988 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.1", 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.2", 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-04-25T07:03:58.989 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.990 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.991 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.992 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-04-25T07:03:58.993 INFO:tasks.workunit.client.0.smithi088.stderr:report 1612529723 2024-04-25T07:03:58.993 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.993 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-04-25T07:03:58.993 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-04-25T07:03:58.993 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.994 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.995 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.996 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.997 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.998 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T07:03:58.999 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.000 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-04-25T07:03:59.001 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 0, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-04-25T07:03:59.002 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-04-25T07:03:59.003 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:03:59.004 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 5, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 3 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 32, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 32, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 3, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 60332, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 97, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T07:03:59.005 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 291, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 97, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 97, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 61, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 61, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T07:03:59.006 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T07:03:59.007 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:03:59.008 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 449, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 449, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 64, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 64, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-04-25T07:03:59.009 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 96, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 84428, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2776, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81589, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314488372, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036092928, 2024-04-25T07:03:59.010 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2842624, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 710933, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83548004 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T07:03:59.011 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 111, 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 111, 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 111000000, 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 111000000 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T07:03:59.012 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 96, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 84428, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2776, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81589, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314488372, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036092928, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2842624, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 710933, 2024-04-25T07:03:59.013 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83548004 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T07:03:59.014 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 111, 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 111, 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 111000000, 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 111000000 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 32, 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T07:03:59.015 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 60332, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 97, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 291, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 97, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 97, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 61, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 61, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:03:59.016 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T07:03:59.017 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T07:03:59.018 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1220608, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 184566, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 449, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 449, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 64, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 64, 2024-04-25T07:03:59.019 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 3 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 257698037771 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 42949672989 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:03:59.020 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 279172874250 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 32 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 19, 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 32, 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:03:59.021 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 13, 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 32, 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 0, 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:59.022 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 5, 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:03:09.362798+0000", 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 92, 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 433, 2024-04-25T07:03:59.023 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-04-25T07:03:59.024 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T07:03:59.024 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 587, 2024-04-25T07:03:59.024 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-04-25T07:03:59.024 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-04-25T07:03:59.024 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:03:59.024 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:03:59.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:03:59.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-04-25T07:03:59.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-04-25T07:03:59.025 INFO:tasks.workunit.client.0.smithi088.stdout:writing non-divergent object existing_4 2024-04-25T07:03:59.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-04-25T07:03:59.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:03:59.315 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:03:59.315 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461367+0000 34'68 73:138 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:03:59.315 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461425+0000 36'68 73:127 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:03:59.315 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 4 0 0 2525 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461376+0000 34'69 73:125 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:03:59.315 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.388898+0000 34'68 73:121 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.028388+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.315 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:22.504678+0000 32'67 73:102 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:03:59.315 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.413544+0000 34'69 73:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107940+0000 0 0 2024-04-25T07:03:59.315 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.416137+0000 34'69 73:121 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107961+0000 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.417118+0000 32'67 73:124 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107930+0000 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 4 0 0 2468 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.409741+0000 34'69 73:140 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:51.774330+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:51:55.962575+0000 4 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 6 0 0 3702 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461763+0000 34'69 73:143 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 6 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.409939+0000 32'67 73:109 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 4 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.134906+0000 36'68 73:120 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:43.547323+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 1 0 0 617 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.413426+0000 34'68 73:140 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:39:49.397810+0000 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389070+0000 34'68 73:121 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 2 0 0 1234 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.415581+0000 34'69 73:150 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:11.505441+0000 2 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 2 0 0 1291 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412516+0000 32'67 73:109 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.409750+0000 34'68 73:129 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:50:17.468836+0000 2 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 3 0 0 1851 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.417049+0000 34'69 73:144 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:21.379974+0000 3 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.134672+0000 34'69 73:116 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 4 0 0 2468 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415631+0000 32'67 73:132 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:11:13.252287+0000 4 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389944+0000 34'68 73:121 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.316 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412573+0000 32'67 73:109 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 1 0 0 617 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.410433+0000 32'67 73:127 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:31:43.477798+0000 1 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461829+0000 34'69 73:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.411323+0000 34'69 73:122 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.071327+0000 34'68 73:137 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:41:54.994652+0000 3 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 4 0 0 2468 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414364+0000 34'68 73:146 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:45:27.211802+0000 4 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415727+0000 32'67 73:124 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107950+0000 0 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414563+0000 35'68 73:123 [2,0] 2 [2,0] 2 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107972+0000 0 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 2 0 0 1234 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.414038+0000 32'67 73:134 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:53:12.034656+0000 2 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414005+0000 34'68 73:120 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107915+0000 0 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.411093+0000 36'68 73:118 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:03:59.317 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:03:59.318 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:03:59.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:03:59.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:03:59.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-04-25T07:03:59.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:03:59.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:03:59.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:03:59.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:03:59.437 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:03:59.437 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:03:59.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:03:59.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:03:59.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:03:59.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:03:59.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:03:59.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:03:59.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:03:59.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:03:59.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:03:59.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:03:59.979 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:03:59.979 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:03:59.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:03:59.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:03:59.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:04:00.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=257698037774 2024-04-25T07:04:00.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 257698037774 2024-04-25T07:04:00.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-257698037774' 2024-04-25T07:04:00.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:04:00.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:04:00.299 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:04:00.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:04:00.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:04:00.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:04:00.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:04:00.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:04:00.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=279172874253 2024-04-25T07:04:00.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 279172874253 2024-04-25T07:04:00.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-257698037774 2-279172874253' 2024-04-25T07:04:00.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:04:00.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-257698037774 2024-04-25T07:04:00.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:04:00.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:04:00.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-257698037774 2024-04-25T07:04:00.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:04:00.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=257698037774 2024-04-25T07:04:00.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 257698037774' 2024-04-25T07:04:00.461 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 257698037774 2024-04-25T07:04:00.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:04:00.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 257698037772 -lt 257698037774 2024-04-25T07:04:00.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:04:01.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:04:01.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:04:02.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 257698037774 -lt 257698037774 2024-04-25T07:04:02.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:04:02.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:04:02.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-279172874253 2024-04-25T07:04:02.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:04:02.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:04:02.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-279172874253 2024-04-25T07:04:02.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=279172874253 2024-04-25T07:04:02.403 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 279172874253 2024-04-25T07:04:02.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 279172874253' 2024-04-25T07:04:02.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:04:02.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 279172874253 -lt 279172874253 2024-04-25T07:04:02.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:04:02.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:02.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:03.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:04:03.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:03.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:03.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:03.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:03.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:03.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:03.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:03.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:03.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:03.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:03.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:04:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:04:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:04:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:04:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:04:04.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:04:04.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:04.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:04.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:04.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:04.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:04.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:04.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:04.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:04.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:04.947 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:04.947 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:05.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:05.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:04:05.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:04:05.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:04:05.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:04:05.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:04:05.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:04:05.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:04:05.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:04:06.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:04:06.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:04:06.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:04:06.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:04:06.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:04:06.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:04:06.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:06.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:06.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:06.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:06.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:06.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:06.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:06.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:06.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:06.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:06.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:07.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:04:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:04:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:04:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:04:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:04:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:04:07.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:04:07.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:04:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:04:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:04:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:04:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:04:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:04:08.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:04:08.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:08.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:08.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:08.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:08.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:08.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:08.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:08.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:08.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:08.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:08.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:09.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:09.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:04:09.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:04:09.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:04:09.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:04:09.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:04:09.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:04:09.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:04:09.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:04:09.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:04:09.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:04:09.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:04:09.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:04:09.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:04:10.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:04:10.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:10.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:10.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:10.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:10.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:10.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:10.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:10.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:10.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:10.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:10.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:11.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:11.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:04:11.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:04:11.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:04:11.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:04:11.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:04:11.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:04:11.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:04:11.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:04:12.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:04:12.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:04:12.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:04:12.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:04:12.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:04:13.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:04:13.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:13.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:13.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:13.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:13.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:13.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:13.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:14.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:14.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:14.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:14.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:04:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:04:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:04:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:04:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:04:14.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:04:14.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:04:14.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:04:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:04:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:04:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:04:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:04:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:04:18.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:04:18.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:18.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:18.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:18.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:18.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:18.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:18.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:18.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:18.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:18.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:18.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:19.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:19.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:04:19.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:04:19.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:04:19.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:04:19.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:04:19.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:04:19.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:04:19.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:04:19.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:04:19.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:04:19.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:04:19.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:04:19.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:04:26.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:04:26.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:26.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:26.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:26.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:26.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:26.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:26.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:26.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:26.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:26.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:26.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:27.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:27.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:04:27.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:04:27.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:04:27.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:04:27.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:04:27.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:04:27.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:04:27.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:04:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:04:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:04:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:04:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:04:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:04:35.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:04:35.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:04:35.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:04:35.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:04:35.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:04:35.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:04:35.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:04:35.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:04:35.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:04:35.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:04:35.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:04:35.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:04:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:04:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:04:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:04:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:04:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:04:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:04:36.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:04:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:04:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:04:36.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:04:36.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:04:36.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:04:36.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "3dc17557-779b-4d76-bc92-991d56bbc098", 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-04-25T07:04:37.155734+0000", 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-04-25T07:04:37.153 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 17 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "OSD_DOWN": { 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:04:37.154 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 osds down", 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "osd.1 (root=default,host=smithi088) is down" 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 97/291 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-04-25T07:04:37.155 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 64 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.0 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.2 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.156 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.4 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.5 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.6 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.7 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.8 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.157 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.9 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.a is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.b is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.c is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.158 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.d is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.e is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.f is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.10 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.11 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.159 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.12 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.13 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.14 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.15 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.160 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.16 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.17 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.18 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.19 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1a is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.161 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1b is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1c is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1d is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1e is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.162 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.163 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T07:04:37.164 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:00:31.110671Z", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T07:00:31.110671Z", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-04-25T07:04:37.165 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-04-25T07:04:37.166 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.167 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 73, 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T07:00:31.992003+0000", 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:03:21.948822+0000", 2024-04-25T07:04:37.168 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-04-25T07:03:18.463446+0000", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-04-25T07:03:20.377028+0000", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5800384, 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "nodown", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "noin", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "noout", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 9, 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:04:37.169 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 3, 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "test", 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T07:02:16.874982+0000", 2024-04-25T07:04:37.170 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 1, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool", 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "on", 2024-04-25T07:04:37.171 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 32, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 19, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 32, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 32, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 32, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "72", 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:04:37.172 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "33", 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 0, 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 0, 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T07:04:37.173 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-04-25T07:04:37.174 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T07:04:37.175 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {}, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 1.7799999713897705, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 1.7799999713897705, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 1.190000057220459, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 1.190000057220459, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.176 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "8db453d6-6a31-457a-ac95-0d30560b9fc3", 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 5, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 41, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 60, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 69, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 51, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:04:37.177 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:04:37.178 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.179 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.180 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 106036612 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6803/106036612", 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6805/106036612", 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6809/106036612", 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6807/106036612", 2024-04-25T07:04:37.181 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "b2d99303-3733-4f44-8f02-eddc6e2cece0", 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 0, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 10, 2024-04-25T07:04:37.182 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 48, 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 52, 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.183 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.184 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:04:37.185 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:04:37.186 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1263838535 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/1263838535", 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/1263838535", 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1263838535", 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1263838535", 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: "exists" 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:04:37.187 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "6160fb33-0967-4995-9dc7-f5ae0db1f5a0", 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 15, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 41, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 65, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 71, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 53, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:04:37.188 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.189 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:04:37.190 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:04:37.191 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2515045402 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/2515045402", 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/2515045402", 2024-04-25T07:04:37.192 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2515045402", 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2515045402", 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:03:05.648204+0000", 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:04:37.193 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:01:04.261621+0000", 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:03:06.267405+0000", 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:01:38.903948+0000", 2024-04-25T07:04:37.194 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:03:06.809043+0000", 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:02:13.056900+0000", 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-04-25T07:04:37.195 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [], 2024-04-25T07:04:37.196 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [], 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:04:37.197 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6804/106036612,v1:127.0.0.1:6805/106036612]", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:04:37.198 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:04:37.199 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:00:38.461379Z", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6802/106036612,v1:127.0.0.1:6803/106036612]", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/106036612,v1:127.0.0.1:6809/106036612]", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/106036612,v1:127.0.0.1:6807/106036612]", 2024-04-25T07:04:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:04:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:04:37.202 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:04:37.202 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.202 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/1263838535,v1:127.0.0.1:6813/1263838535]", 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:04:37.203 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:04:37.204 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:01:11.931407Z", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/1263838535,v1:127.0.0.1:6811/1263838535]", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1263838535,v1:127.0.0.1:6817/1263838535]", 2024-04-25T07:04:37.205 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1263838535,v1:127.0.0.1:6815/1263838535]", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:04:37.206 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/2515045402,v1:127.0.0.1:6813/2515045402]", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:04:37.207 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:04:37.208 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:01:46.288989Z", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/2515045402,v1:127.0.0.1:6811/2515045402]", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2515045402,v1:127.0.0.1:6817/2515045402]", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2515045402,v1:127.0.0.1:6815/2515045402]", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:04:37.209 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-04-25T07:04:37.210 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [] 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 61 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 65 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.211 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.212 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-04-25T07:04:37.212 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 73, 2024-04-25T07:04:37.212 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-04-25T07:04:37.212 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-04-25T07:04:37.212 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.212 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:04:37.212 INFO:tasks.workunit.client.0.smithi088.stderr:report 3038690632 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.1", 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.2", 2024-04-25T07:04:37.213 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-04-25T07:04:37.214 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.215 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.216 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:04:37.217 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.218 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:04:37.219 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:04:37.220 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.221 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:04:37.221 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.222 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:04:37.223 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-04-25T07:04:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:04:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 0, 2024-04-25T07:04:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-04-25T07:04:37.228 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-04-25T07:04:37.229 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 5, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 3 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 32, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 32, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 3, 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-04-25T07:04:37.230 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 60682, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 97, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 291, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 97, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 97, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 62, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 62, 2024-04-25T07:04:37.231 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T07:04:37.232 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:04:37.233 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 450, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 450, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 64, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 64, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T07:04:37.234 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 96, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 84428, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2776, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81589, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314488372, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036092928, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2842624, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 711633, 2024-04-25T07:04:37.235 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83548004 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 111, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 111, 2024-04-25T07:04:37.236 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 111000000, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 111000000 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 96, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:04:37.237 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T07:04:37.238 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 84428, 2024-04-25T07:04:37.238 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2776, 2024-04-25T07:04:37.238 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T07:04:37.238 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81589, 2024-04-25T07:04:37.238 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314488372, 2024-04-25T07:04:37.238 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T07:04:37.238 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T07:04:37.238 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036092928, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2842624, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 711633, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83548004 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T07:04:37.239 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 111, 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 111, 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 111000000, 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 111000000 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 32, 2024-04-25T07:04:37.240 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 60682, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 97, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 291, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 97, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 97, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 62, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 62, 2024-04-25T07:04:37.241 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T07:04:37.242 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1220608, 2024-04-25T07:04:37.243 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 185266, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 450, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 450, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 64, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 64, 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 3 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 257698037781 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 42949672989 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 279172874260 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-04-25T07:04:37.245 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 32 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 19, 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 32, 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:04:37.246 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 13, 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 32, 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 0, 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:04:37.247 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 7, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:04:19.388782+0000", 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 111, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 501, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 674, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:04:37.248 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:04:37.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:04:37.249 INFO:tasks.workunit.client.0.smithi088.stdout:delay recovery 2024-04-25T07:04:37.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-04-25T07:04:37.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:04:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:04:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461367+0000 34'68 73:138 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:04:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461425+0000 36'68 73:127 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:04:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 4 0 0 2525 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461376+0000 34'69 73:125 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.388898+0000 34'68 73:121 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.028388+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:22.504678+0000 32'67 73:102 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.413544+0000 34'69 73:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107940+0000 0 0 2024-04-25T07:04:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.416137+0000 34'69 73:121 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107961+0000 0 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.417118+0000 32'67 73:124 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107930+0000 0 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 4 0 0 2468 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.409741+0000 34'69 73:140 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:51.774330+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:51:55.962575+0000 4 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 6 0 0 3702 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461763+0000 34'69 73:143 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 6 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.409939+0000 32'67 73:109 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 4 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.134906+0000 36'68 73:120 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:43.547323+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 1 0 0 617 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.413426+0000 34'68 73:140 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:39:49.397810+0000 0 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389070+0000 34'68 73:121 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 2 0 0 1234 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.415581+0000 34'69 73:150 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:11.505441+0000 2 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 2 0 0 1291 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412516+0000 32'67 73:109 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.409750+0000 34'68 73:129 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:50:17.468836+0000 2 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 3 0 0 1851 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.417049+0000 34'69 73:144 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:21.379974+0000 3 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 4 0 0 2876 0 0 16 54 16 active+undersized+degraded 2024-04-25T07:03:22.134672+0000 73'70 73:117 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.548 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 4 0 0 2468 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415631+0000 32'67 73:132 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:11:13.252287+0000 4 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389944+0000 34'68 73:121 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412573+0000 32'67 73:109 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 1 0 0 617 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.410433+0000 32'67 73:127 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:31:43.477798+0000 1 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461829+0000 34'69 73:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.411323+0000 34'69 73:122 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.071327+0000 34'68 73:137 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:41:54.994652+0000 3 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 4 0 0 2468 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414364+0000 34'68 73:146 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:45:27.211802+0000 4 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415727+0000 32'67 73:124 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107950+0000 0 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414563+0000 35'68 73:123 [2,0] 2 [2,0] 2 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107972+0000 0 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 2 0 0 1234 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.414038+0000 32'67 73:134 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:53:12.034656+0000 2 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414005+0000 34'68 73:120 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107915+0000 0 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.411093+0000 36'68 73:118 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:04:37.549 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:04:37.550 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:04:37.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:04:37.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-04-25T07:04:37.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:04:37.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:04:37.563 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:04:37.563 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:04:37.563 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:04:37.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.0.asok 2024-04-25T07:04:37.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:04:37.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:04:37.662 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:04:37.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:04:37.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-04-25T07:04:37.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:04:37.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:04:37.675 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:04:37.675 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:04:37.675 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:04:37.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.2.asok 2024-04-25T07:04:37.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:04:37.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:04:37.788 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:04:37.801 INFO:tasks.workunit.client.0.smithi088.stdout:revive divergent 1 2024-04-25T07:04:37.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-04-25T07:04:37.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:04:38.193 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461367+0000 34'68 73:138 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461425+0000 36'68 73:127 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 4 0 0 2525 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461376+0000 34'69 73:125 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.388898+0000 34'68 73:121 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.028388+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:22.504678+0000 32'67 73:102 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.413544+0000 34'69 73:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107940+0000 0 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.416137+0000 34'69 73:121 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107961+0000 0 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.417118+0000 32'67 73:124 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107930+0000 0 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 4 0 0 2468 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.409741+0000 34'69 73:140 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:51.774330+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:51:55.962575+0000 4 0 2024-04-25T07:04:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 6 0 0 3702 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461763+0000 34'69 73:143 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 no scrub is scheduled 6 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.409939+0000 32'67 73:109 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 4 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.134906+0000 36'68 73:120 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:43.547323+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 1 0 0 617 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.413426+0000 34'68 73:140 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:39:49.397810+0000 0 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389070+0000 34'68 73:121 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 2 0 0 1234 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.415581+0000 34'69 73:150 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:11.505441+0000 2 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 2 0 0 1291 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412516+0000 32'67 73:109 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.409750+0000 34'68 73:129 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:50:17.468836+0000 2 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 3 0 0 1851 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.417049+0000 34'69 73:144 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:21.379974+0000 3 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 4 0 0 2876 0 0 16 54 16 active+undersized+degraded 2024-04-25T07:03:22.134672+0000 73'70 73:117 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 4 0 0 2468 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415631+0000 32'67 73:132 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:11:13.252287+0000 4 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389944+0000 34'68 73:121 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412573+0000 32'67 73:109 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 1 0 0 617 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.410433+0000 32'67 73:127 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:31:43.477798+0000 1 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461829+0000 34'69 73:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.411323+0000 34'69 73:122 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.071327+0000 34'68 73:137 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:41:54.994652+0000 3 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 4 0 0 2468 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414364+0000 34'68 73:146 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:45:27.211802+0000 4 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415727+0000 32'67 73:124 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107950+0000 0 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414563+0000 35'68 73:123 [2,0] 2 [2,0] 2 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107972+0000 0 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 2 0 0 1234 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.414038+0000 32'67 73:134 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:53:12.034656+0000 2 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414005+0000 34'68 73:120 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:03:27.107915+0000 0 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.411093+0000 36'68 73:118 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:04:38.196 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:04:38.197 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:04:38.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-04-25T07:04:38.876 INFO:tasks.workunit.client.0.smithi088.stderr:noup is set 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:04:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:04:38.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:04:38.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:04:38.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:04:38.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:04:38.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:04:38.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:04:38.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:04:38.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:04:38.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:04:38.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:04:38.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:04:38.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:04:38.900 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:04:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:04:38.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:04:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:04:38.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:04:38.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:04:38.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:04:38.940 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:38.942+0000 7f5d8965d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:38.943 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:38.942+0000 7f5d8965d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:38.944 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:38.946+0000 7f5d8965d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:39.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:04:39.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-04-25T07:04:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:39.494+0000 7f5d8965d7c0 -1 Falling back to public interface 2024-04-25T07:04:42.509 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:42.510+0000 7f5d8965d7c0 -1 osd.1 49 log_to_monitors true 2024-04-25T07:04:44.378 INFO:tasks.workunit.client.0.smithi088.stdout:delay recovery divergent 2024-04-25T07:04:44.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-04-25T07:04:44.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:04:44.796 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:04:44.796 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461367+0000 34'68 75:142 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:29.895472+0000 3 0 2024-04-25T07:04:44.796 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461425+0000 36'68 75:131 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:06:00.434517+0000 2 0 2024-04-25T07:04:44.796 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 4 0 0 2525 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461376+0000 34'69 75:129 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184525+0000 0 0 2024-04-25T07:04:44.796 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.388898+0000 34'68 75:124 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.028388+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:39.074543+0000 0 0 2024-04-25T07:04:44.796 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:22.504678+0000 32'67 75:106 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184564+0000 0 0 2024-04-25T07:04:44.796 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.413544+0000 34'69 75:135 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184408+0000 0 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.416137+0000 34'69 75:125 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184449+0000 0 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.417118+0000 32'67 75:128 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184388+0000 0 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 4 0 0 2468 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.409741+0000 34'69 75:143 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:51.774330+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:51:55.962575+0000 4 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 6 0 0 3702 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461763+0000 34'69 75:147 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:21:42.534421+0000 6 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.409939+0000 32'67 75:112 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:39.074652+0000 0 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 4 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.134906+0000 36'68 75:124 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:43.547323+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184506+0000 0 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 1 0 0 617 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.413426+0000 34'68 75:143 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:39:49.397810+0000 0 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389070+0000 34'68 75:124 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:39.074568+0000 0 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 2 0 0 1234 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.415581+0000 34'69 75:154 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:11.505441+0000 2 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 2 0 0 1291 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412516+0000 32'67 75:112 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:39.074691+0000 0 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.409750+0000 34'68 75:132 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:50:17.468836+0000 2 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 3 0 0 1851 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.417049+0000 34'69 75:148 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:21.379974+0000 3 0 2024-04-25T07:04:44.797 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 4 0 0 2876 0 0 16 54 16 active+undersized+degraded 2024-04-25T07:03:22.134672+0000 73'70 75:121 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184487+0000 0 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 4 0 0 2468 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415631+0000 32'67 75:136 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:11:13.252287+0000 4 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389944+0000 34'68 75:124 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:39.074589+0000 0 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412573+0000 32'67 75:112 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:39.074630+0000 0 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 1 0 0 617 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.410433+0000 32'67 75:130 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:31:43.477798+0000 1 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461829+0000 34'69 75:135 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184543+0000 0 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.411323+0000 34'69 75:125 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:39.074672+0000 0 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.071327+0000 34'68 75:141 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:41:54.994652+0000 3 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 4 0 0 2468 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414364+0000 34'68 75:149 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:45:27.211802+0000 4 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415727+0000 32'67 75:128 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184427+0000 0 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414563+0000 35'68 75:127 [2,0] 2 [2,0] 2 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184468+0000 0 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 2 0 0 1234 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.414038+0000 32'67 75:138 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:53:12.034656+0000 2 0 2024-04-25T07:04:44.798 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414005+0000 34'68 75:124 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184364+0000 0 0 2024-04-25T07:04:44.799 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.411093+0000 36'68 75:121 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:39.074611+0000 0 0 2024-04-25T07:04:44.799 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:04:44.799 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:04:44.799 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:04:44.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-04-25T07:04:44.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:04:44.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:04:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:04:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:04:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:04:44.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.1.asok 2024-04-25T07:04:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:04:44.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:04:44.922 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:04:44.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-04-25T07:04:45.623 INFO:tasks.workunit.client.0.smithi088.stderr:noup is unset 2024-04-25T07:04:45.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-04-25T07:04:45.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:04:45.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:04:45.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:04:45.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:04:45.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:45.642 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:04:45.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:04:45.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:45.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:04:46.208 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 60 up_thru 69 down_at 51 last_clean_interval [5,41) [v2:127.0.0.1:6802/106036612,v1:127.0.0.1:6803/106036612] [v2:127.0.0.1:6804/106036612,v1:127.0.0.1:6805/106036612] exists,up 8db453d6-6a31-457a-ac95-0d30560b9fc3 2024-04-25T07:04:46.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:04:46.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:04:46.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:04:46.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-04-25T07:04:46.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:04:46.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:04:46.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:04:46.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:04:46.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:46.210 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:04:46.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:04:46.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:46.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:46.739 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 52 last_clean_interval [10,49) [v2:127.0.0.1:6818/1469921032,v1:127.0.0.1:6819/1469921032] [v2:127.0.0.1:6820/1469921032,v1:127.0.0.1:6821/1469921032] exists,up b2d99303-3733-4f44-8f02-eddc6e2cece0 2024-04-25T07:04:46.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:04:46.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:04:46.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:04:46.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-04-25T07:04:46.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:04:46.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:04:46.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:04:46.741 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:04:46.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:04:46.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:46.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:04:46.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:04:46.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:47.329 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 65 up_thru 79 down_at 53 last_clean_interval [15,41) [v2:127.0.0.1:6810/2515045402,v1:127.0.0.1:6811/2515045402] [v2:127.0.0.1:6812/2515045402,v1:127.0.0.1:6813/2515045402] exists,up 6160fb33-0967-4995-9dc7-f5ae0db1f5a0 2024-04-25T07:04:47.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:04:47.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:04:47.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:04:47.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461367+0000 34'68 76:144 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:29.895472+0000 3 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461425+0000 36'68 76:133 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:06:00.434517+0000 2 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 4 0 0 2525 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461376+0000 34'69 76:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184525+0000 0 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.388898+0000 34'68 76:126 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.028388+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015190+0000 0 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:22.504678+0000 32'67 76:108 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184564+0000 0 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.413544+0000 34'69 76:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184408+0000 0 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.416137+0000 34'69 76:127 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184449+0000 0 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.417118+0000 32'67 76:130 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184388+0000 0 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 4 0 0 2468 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.409741+0000 34'69 76:145 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:51.774330+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:51:55.962575+0000 4 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 6 0 0 3702 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461763+0000 34'69 76:149 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:21:42.534421+0000 6 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.409939+0000 32'67 76:114 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015300+0000 0 0 2024-04-25T07:04:47.739 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 4 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.134906+0000 36'68 76:126 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:43.547323+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184506+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 1 0 0 617 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.413426+0000 34'68 76:145 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:39:49.397810+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389070+0000 34'68 76:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015216+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 2 0 0 1234 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.415581+0000 34'69 76:156 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:11.505441+0000 2 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 2 0 0 1291 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412516+0000 32'67 76:114 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015342+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.409750+0000 34'68 76:134 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:50:17.468836+0000 2 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 3 0 0 1851 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.417049+0000 34'69 76:150 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:21.379974+0000 3 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 4 0 0 2876 0 0 16 54 16 active+undersized+degraded 2024-04-25T07:03:22.134672+0000 73'70 76:123 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184487+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 4 0 0 2468 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415631+0000 32'67 76:138 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:11:13.252287+0000 4 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389944+0000 34'68 76:126 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015237+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412573+0000 32'67 76:114 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015280+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 1 0 0 617 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.410433+0000 32'67 76:132 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:31:43.477798+0000 1 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461829+0000 34'69 76:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184543+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.411323+0000 34'69 76:127 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015320+0000 0 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.071327+0000 34'68 76:143 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:41:54.994652+0000 3 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 4 0 0 2468 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414364+0000 34'68 76:151 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:45:27.211802+0000 4 0 2024-04-25T07:04:47.740 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415727+0000 32'67 76:130 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184427+0000 0 0 2024-04-25T07:04:47.741 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414563+0000 35'68 76:129 [2,0] 2 [2,0] 2 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184468+0000 0 0 2024-04-25T07:04:47.741 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 2 0 0 1234 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.414038+0000 32'67 76:140 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:53:12.034656+0000 2 0 2024-04-25T07:04:47.741 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414005+0000 34'68 76:126 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184364+0000 0 0 2024-04-25T07:04:47.741 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.411093+0000 36'68 76:123 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015260+0000 0 0 2024-04-25T07:04:47.741 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:04:47.741 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:04:47.741 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:04:47.754 INFO:tasks.workunit.client.0.smithi088.stdout:wait for peering 2024-04-25T07:04:47.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-04-25T07:04:47.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:04:48.140 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:04:48.140 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461367+0000 34'68 76:144 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:29.895472+0000 3 0 2024-04-25T07:04:48.140 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461425+0000 36'68 76:133 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:06:00.434517+0000 2 0 2024-04-25T07:04:48.140 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 4 0 0 2525 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461376+0000 34'69 76:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184525+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.388898+0000 34'68 76:126 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.028388+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015190+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:22.504678+0000 32'67 76:108 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184564+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.413544+0000 34'69 76:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184408+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.416137+0000 34'69 76:127 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184449+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.417118+0000 32'67 76:130 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184388+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 4 0 0 2468 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.409741+0000 34'69 76:145 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:51.774330+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:51:55.962575+0000 4 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 6 0 0 3702 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461763+0000 34'69 76:149 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:21:42.534421+0000 6 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.409939+0000 32'67 76:114 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015300+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 4 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.134906+0000 36'68 76:126 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:43.547323+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184506+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 1 0 0 617 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.413426+0000 34'68 76:145 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:39:49.397810+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389070+0000 34'68 76:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015216+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 2 0 0 1234 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.415581+0000 34'69 76:156 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:11.505441+0000 2 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 2 0 0 1291 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412516+0000 32'67 76:114 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015342+0000 0 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.409750+0000 34'68 76:134 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:50:17.468836+0000 2 0 2024-04-25T07:04:48.141 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 3 0 0 1851 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.417049+0000 34'69 76:150 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:21.379974+0000 3 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 4 0 0 2876 0 0 16 54 16 active+undersized+degraded 2024-04-25T07:03:22.134672+0000 73'70 76:123 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184487+0000 0 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 4 0 0 2468 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415631+0000 32'67 76:138 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:11:13.252287+0000 4 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389944+0000 34'68 76:126 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015237+0000 0 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412573+0000 32'67 76:114 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015280+0000 0 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 1 0 0 617 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.410433+0000 32'67 76:132 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:31:43.477798+0000 1 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461829+0000 34'69 76:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184543+0000 0 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.411323+0000 34'69 76:127 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015320+0000 0 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.071327+0000 34'68 76:143 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:41:54.994652+0000 3 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 4 0 0 2468 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414364+0000 34'68 76:151 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:45:27.211802+0000 4 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415727+0000 32'67 76:130 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184427+0000 0 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414563+0000 35'68 76:129 [2,0] 2 [2,0] 2 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184468+0000 0 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 2 0 0 1234 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.414038+0000 32'67 76:140 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:53:12.034656+0000 2 0 2024-04-25T07:04:48.142 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414005+0000 34'68 76:126 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184364+0000 0 0 2024-04-25T07:04:48.143 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.411093+0000 36'68 76:123 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015260+0000 0 0 2024-04-25T07:04:48.143 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:04:48.143 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:04:48.143 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:04:48.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-04-25T07:04:48.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-04-25T07:04:48.289 INFO:tasks.workunit.client.0.smithi088.stdout:killing divergent 1 2024-04-25T07:04:48.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:04:48.678 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:04:48.678 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461367+0000 34'68 76:144 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:29.895472+0000 3 0 2024-04-25T07:04:48.678 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.461425+0000 36'68 76:133 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:06:00.434517+0000 2 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 4 0 0 2525 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461376+0000 34'69 76:131 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184525+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.388898+0000 34'68 76:126 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.028388+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015190+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:22.504678+0000 32'67 76:108 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184564+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.413544+0000 34'69 76:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184408+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.416137+0000 34'69 76:127 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184449+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.417118+0000 32'67 76:130 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184388+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 4 0 0 2468 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.409741+0000 34'69 76:145 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:51.774330+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:51:55.962575+0000 4 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 6 0 0 3702 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461763+0000 34'69 76:149 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:21:42.534421+0000 6 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.409939+0000 32'67 76:114 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015300+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 4 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.134906+0000 36'68 76:126 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:43.547323+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184506+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 1 0 0 617 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.413426+0000 34'68 76:145 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:39:49.397810+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389070+0000 34'68 76:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015216+0000 0 0 2024-04-25T07:04:48.679 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 2 0 0 1234 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.415581+0000 34'69 76:156 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:11.505441+0000 2 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 2 0 0 1291 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412516+0000 32'67 76:114 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015342+0000 0 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.409750+0000 34'68 76:134 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:50:17.468836+0000 2 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 3 0 0 1851 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.417049+0000 34'69 76:150 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:52:21.379974+0000 3 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 4 0 0 2876 0 0 16 54 16 active+undersized+degraded 2024-04-25T07:03:22.134672+0000 73'70 76:123 [2,0] 2 [2,0] 2 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184487+0000 0 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 4 0 0 2468 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415631+0000 32'67 76:138 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:11:13.252287+0000 4 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.389944+0000 34'68 76:126 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015237+0000 0 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.412573+0000 32'67 76:114 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015280+0000 0 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 1 0 0 617 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.410433+0000 32'67 76:132 [0,2] 0 [0,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:31:43.477798+0000 1 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:22.461829+0000 34'69 76:137 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184543+0000 0 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 4 0 0 2526 0 0 15 54 15 active+undersized+degraded 2024-04-25T07:03:21.411323+0000 34'69 76:127 [0,2] 0 [0,2] 0 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015320+0000 0 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 3 0 0 1851 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:22.071327+0000 34'68 76:143 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:41:54.994652+0000 3 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 4 0 0 2468 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414364+0000 34'68 76:151 [0,2] 0 [0,2] 0 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:45:27.211802+0000 4 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 2 0 0 1292 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.415727+0000 32'67 76:130 [2,0] 2 [2,0] 2 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184427+0000 0 0 2024-04-25T07:04:48.680 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414563+0000 35'68 76:129 [2,0] 2 [2,0] 2 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184468+0000 0 0 2024-04-25T07:04:48.681 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 2 0 0 1234 0 0 13 54 13 active+undersized+degraded 2024-04-25T07:03:21.414038+0000 32'67 76:140 [2,0] 2 [2,0] 2 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:53:12.034656+0000 2 0 2024-04-25T07:04:48.681 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.414005+0000 34'68 76:126 [2,0] 2 [2,0] 2 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:44.184364+0000 0 0 2024-04-25T07:04:48.681 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 0 3 0 0 1909 0 0 14 54 14 active+undersized+degraded 2024-04-25T07:03:21.411093+0000 36'68 76:123 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:02:16.902637+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:04:45.015260+0000 0 0 2024-04-25T07:04:48.681 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:04:48.681 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:04:48.681 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:04:48.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:04:48.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:04:48.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:04:48.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:04:48.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:04:48.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:04:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:04:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.26017.out 2024-04-25T07:04:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.26017.out 2024-04-25T07:04:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:04:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:04:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:04:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:04:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:04:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.26017.out 2024-04-25T07:04:49.949 INFO:tasks.workunit.client.0.smithi088.stderr:Exporting 1.1b info 1.1b( v 36'68 lc 0'0 (31'54,36'68] local-lis/les=79/80 n=3 ec=33/23 lis/c=79/39 les/c/f=80/40/0 sis=79) 2024-04-25T07:04:49.950 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:04:49.959 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-04-25T07:04:49.960 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dd677bd3:::existing_35:head# 2024-04-25T07:04:49.967 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-04-25T07:04:49.967 INFO:tasks.workunit.client.0.smithi088.stderr:Export successful 2024-04-25T07:04:49.968 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T07:04:49.968 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T07:04:49.968 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-04-25T07:04:50.116 INFO:tasks.workunit.client.0.smithi088.stderr:Remove successful 2024-04-25T07:04:50.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.26017.out 2024-04-25T07:04:50.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:04:50.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:04:50.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:04:50.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:04:50.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:04:50.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.26017.out 2024-04-25T07:04:51.297 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e81 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-04-25T07:04:51.297 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.1b 2024-04-25T07:04:51.297 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:04:51.297 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:04:51.352 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-04-25T07:04:51.352 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:04:51.385 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd677bd3:::existing_35:head# 2024-04-25T07:04:51.385 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:04:51.418 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-04-25T07:04:51.418 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T07:04:51.452 INFO:tasks.workunit.client.0.smithi088.stdout:write_pg epoch 80 info 1.1b( v 36'68 lc 0'0 (31'54,36'68] local-lis/les=79/80 n=3 ec=33/23 lis/c=79/39 les/c/f=80/40/0 sis=79) 2024-04-25T07:04:51.502 INFO:tasks.workunit.client.0.smithi088.stdout:Import successful 2024-04-25T07:04:52.069 INFO:tasks.workunit.client.0.smithi088.stdout:reviving divergent 1 2024-04-25T07:04:52.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-04-25T07:04:52.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:46.639545+0000 34'68 81:159 [2,0,1] 2 [2,0,1] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:22:08.257372+0000 3 0 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:46.650781+0000 36'68 81:143 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:39:18.039175+0000 2 0 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 0 0 0 2525 0 0 15 54 15 active+clean 2024-04-25T07:04:46.639151+0000 34'69 81:146 [2,1,0] 2 [2,1,0] 2 34'69 2024-04-25T07:02:55.069431+0000 34'69 2024-04-25T07:02:55.069431+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:04:46.646051+0000 34'68 81:146 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:57.636504+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 3 0 0 0 0 1851 0 0 13 54 13 active+clean 2024-04-25T07:04:46.651425+0000 32'67 81:124 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:43:47.158998+0000 3 0 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 4 1 1 0 0 2526 0 0 15 54 15 active+recovery_wait+degraded 2024-04-25T07:04:46.649244+0000 34'69 81:158 [1,2,0] 1 [1,2,0] 1 41'70 2024-04-25T07:03:04.585868+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:04:46.648358+0000 34'69 81:168 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.481 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 0 0 0 1292 0 0 13 54 13 active+clean 2024-04-25T07:04:46.645041+0000 32'67 81:147 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:59.547469+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 4 0 0 0 0 2468 0 0 15 54 15 active+clean 2024-04-25T07:04:46.644038+0000 34'69 81:149 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:03:01.636353+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:24.114611+0000 4 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 0 0 0 3702 0 0 15 54 15 active+clean 2024-04-25T07:04:46.643255+0000 34'69 81:164 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:21:22.473139+0000 6 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 0 0 0 1292 0 0 13 54 13 active+clean 2024-04-25T07:04:46.648862+0000 32'67 81:126 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:42.576422+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:04:46.641991+0000 36'68 81:145 [1,2,0] 1 [1,2,0] 1 36'68 2024-04-25T07:02:58.637424+0000 36'68 2024-04-25T07:02:58.637424+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 0 0 0 617 0 0 14 54 14 active+clean 2024-04-25T07:04:47.080967+0000 34'68 81:159 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:47:17.895723+0000 0 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:04:46.643247+0000 34'68 81:148 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:55.659162+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 0 0 0 1234 0 0 15 54 15 active+clean 2024-04-25T07:04:46.640151+0000 34'69 81:171 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:10:43.659852+0000 2 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 0 0 0 1291 0 0 13 54 13 active+clean 2024-04-25T07:04:46.651873+0000 32'67 81:136 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:48.638881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:46.650135+0000 34'68 81:128 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:05:46.959719+0000 2 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:04:46.639083+0000 34'69 81:165 [2,1,0] 2 [2,1,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:07:27.045688+0000 3 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 1 1 0 0 2876 0 0 16 54 16 active+recovery_wait+degraded 2024-04-25T07:04:46.647268+0000 73'70 81:155 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:02:50.801258+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:52.482 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 0 0 0 2468 0 0 13 54 13 active+clean 2024-04-25T07:04:46.639274+0000 32'67 81:153 [2,0,1] 2 [2,0,1] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:02:26.919226+0000 4 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:04:47.080363+0000 34'68 81:140 [0,2,1] 0 [0,2,1] 0 34'68 2024-04-25T07:02:49.578753+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 2 0 1 0 0 1292 0 0 13 54 13 active+recovery_wait+degraded 2024-04-25T07:04:47.155569+0000 32'67 81:127 [0,1,2] 0 [0,1,2] 0 41'68 2024-04-25T07:02:55.991960+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 0 0 0 617 0 0 13 54 13 active+clean 2024-04-25T07:04:47.079546+0000 32'67 81:146 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:39:06.905707+0000 1 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:04:46.649097+0000 34'69 81:152 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:03:00.588452+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:04:46.648081+0000 34'69 81:143 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:02:52.731416+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:46.638810+0000 34'68 81:158 [2,0,1] 2 [2,0,1] 2 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:23:28.439993+0000 3 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 0 0 0 2468 0 0 14 54 14 active+clean 2024-04-25T07:04:46.638654+0000 34'68 81:143 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:43:20.977150+0000 4 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 0 0 0 1292 0 0 13 54 13 active+clean 2024-04-25T07:04:46.651245+0000 32'67 81:142 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:56.558814+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:04:46.648298+0000 35'68 81:167 [1,2,0] 1 [1,2,0] 1 35'68 2024-04-25T07:02:50.576546+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 0 0 0 1292 0 0 13 54 13 active+clean 2024-04-25T07:04:46.651810+0000 32'67 81:128 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:52.125228+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:04:52.483 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 4 0 0 0 0 2468 0 0 14 54 14 active+clean 2024-04-25T07:04:46.645307+0000 34'68 81:125 [1,2,0] 1 [1,2,0] 1 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:22:35.820887+0000 4 0 2024-04-25T07:04:52.484 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 3 1 1 0 0 1909 0 0 14 54 14 active+recovering+degraded 2024-04-25T07:04:46.654325+0000 36'68 81:137 [1,0,2] 1 [1,0,2] 1 36'68 2024-04-25T07:02:54.593475+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:04:52.484 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:04:52.484 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:04:52.484 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:04:52.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-04-25T07:04:52.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:04:52.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:04:52.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:04:52.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:04:52.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:04:52.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:04:52.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:04:52.500 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:04:52.500 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:04:52.500 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:04:52.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:04:52.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:04:52.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:04:52.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:04:52.502 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:04:52.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:04:52.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:04:52.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:04:52.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:04:52.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:04:52.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:04:52.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:04:52.539 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:52.538+0000 7f69a544f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:52.541 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:52.542+0000 7f69a544f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:52.542 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:52.542+0000 7f69a544f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:04:52.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:04:52.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:04:52.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:04:52.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:04:52.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:04:52.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:52.941 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:04:52.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:04:52.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:52.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:53.105 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:53.106+0000 7f69a544f7c0 -1 Falling back to public interface 2024-04-25T07:04:53.357 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 52 last_clean_interval [10,49) [v2:127.0.0.1:6818/1469921032,v1:127.0.0.1:6819/1469921032] [v2:127.0.0.1:6820/1469921032,v1:127.0.0.1:6821/1469921032] exists,up b2d99303-3733-4f44-8f02-eddc6e2cece0 2024-04-25T07:04:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:04:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:04:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:04:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-04-25T07:04:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:04:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:04:53.358 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:04:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:04:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:04:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:04:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:04:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:04:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:04:53.819 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 52 last_clean_interval [10,49) [v2:127.0.0.1:6818/1469921032,v1:127.0.0.1:6819/1469921032] [v2:127.0.0.1:6820/1469921032,v1:127.0.0.1:6821/1469921032] exists,up b2d99303-3733-4f44-8f02-eddc6e2cece0 2024-04-25T07:04:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:04:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:04:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:04:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-04-25T07:04:55.874 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:04:55.874+0000 7f69a544f7c0 -1 osd.1 81 log_to_monitors true 2024-04-25T07:05:13.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-04-25T07:05:13.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:05:13.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:05:13.823 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:13.823 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:13.823 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:13.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.1.asok 2024-04-25T07:05:13.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:05:13.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T07:05:13.940 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:05:13.940 INFO:tasks.workunit.client.0.smithi088.stdout: "ops": [], 2024-04-25T07:05:13.940 INFO:tasks.workunit.client.0.smithi088.stdout: "num_ops": 0 2024-04-25T07:05:13.940 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:05:13.952 INFO:tasks.workunit.client.0.smithi088.stdout:allowing recovery 2024-04-25T07:05:13.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-04-25T07:05:13.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:57.662749+0000 34'68 92:187 [2,0,1] 2 [2,0,1] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:59:13.669885+0000 3 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:57.666128+0000 36'68 92:168 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:02:11.818475+0000 2 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 0 0 0 2468 0 0 15 54 15 active+clean 2024-04-25T07:04:58.399989+0000 34'69 92:182 [2,1,0] 2 [2,1,0] 2 34'69 2024-04-25T07:04:58.399782+0000 34'69 2024-04-25T07:02:55.069431+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:00:12.735488+0000 4 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:05:02.848314+0000 34'68 92:179 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:05:02.848111+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:11:50.597271+0000 3 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 3 0 0 0 0 1851 0 0 13 54 13 active+clean 2024-04-25T07:04:57.664849+0000 32'67 92:148 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:22:31.276365+0000 3 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 5 0 0 0 0 3085 0 0 15 54 15 active+clean 2024-04-25T07:05:06.031659+0000 34'69 92:196 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:05:06.031443+0000 34'69 2024-04-25T07:05:06.031443+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:00:44.686488+0000 5 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:04:57.661555+0000 34'69 92:193 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:05:04.836530+0000 32'67 92:180 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:05:04.836327+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:27.761509+0000 2 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 5 0 0 0 0 3085 0 0 16 54 16 active+clean 2024-04-25T07:04:57.673689+0000 81'70 92:176 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:03:01.636353+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:25:16.668383+0000 4 0 2024-04-25T07:05:14.351 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 0 0 0 3702 0 0 15 54 15 active+clean 2024-04-25T07:04:57.638433+0000 34'69 92:192 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:25:53.689638+0000 6 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:05:08.955855+0000 32'67 92:159 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:05:08.955656+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:13:07.564476+0000 2 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:04:57.664612+0000 36'68 92:170 [1,2,0] 1 [1,2,0] 1 36'68 2024-04-25T07:02:58.637424+0000 36'68 2024-04-25T07:02:58.637424+0000 0 0 queued for scrub 0 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 0 0 0 617 0 0 14 54 14 active+clean 2024-04-25T07:04:57.639260+0000 34'68 94:194 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:00:18.042255+0000 0 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:05:06.925647+0000 34'68 92:181 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:05:06.925446+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:51:35.244414+0000 3 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 0 0 0 1234 0 0 15 54 15 active+clean 2024-04-25T07:04:57.661863+0000 34'69 92:199 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:29:26.471755+0000 2 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:05:03.881404+0000 32'67 92:169 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:05:03.881196+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:06:18.017832+0000 2 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:57.667762+0000 34'68 92:153 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:55:23.652870+0000 2 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:04:57.641049+0000 34'69 92:193 [2,1,0] 2 [2,1,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:00:25.428703+0000 3 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 0 0 0 2818 0 0 16 54 16 active+clean 2024-04-25T07:04:59.919507+0000 73'70 92:193 [1,2,0] 1 [1,2,0] 1 73'70 2024-04-25T07:04:59.919281+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:47:53.870417+0000 4 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 0 0 0 2468 0 0 13 54 13 active+clean 2024-04-25T07:04:57.639346+0000 32'67 92:181 [2,0,1] 2 [2,0,1] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:16:13.802342+0000 4 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:05:03.148568+0000 34'68 94:183 [0,2,1] 0 [0,2,1] 0 34'68 2024-04-25T07:05:03.148374+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:48:10.936244+0000 3 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 3 0 0 0 0 1851 0 0 13 54 13 active+clean+inconsistent 2024-04-25T07:05:02.196729+0000 32'67 94:170 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:05:02.196381+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:23:56.305176+0000 3 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 0 0 0 617 0 0 13 54 13 active+clean 2024-04-25T07:04:57.615521+0000 32'67 94:181 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:39:58.170882+0000 1 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:05:07.912992+0000 34'69 92:190 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:05:07.912761+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:34:36.907655+0000 3 0 2024-04-25T07:05:14.352 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:05:04.384861+0000 34'69 92:129 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:05:04.384533+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:21:03.674049+0000 3 0 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:05:06.332237+0000 34'68 92:116 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 0 0 0 2468 0 0 14 54 14 active+clean 2024-04-25T07:04:57.668312+0000 34'68 92:168 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:44:07.604489+0000 4 0 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 0 0 0 1292 0 0 13 54 13 peering 2024-04-25T07:03:20.498757+0000 32'67 94:95 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:05:09.961881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:05:10.937769+0000 35'68 92:200 [1,2,0] 1 [1,2,0] 1 35'68 2024-04-25T07:05:10.937567+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:25:21.543892+0000 3 0 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:05:01.940676+0000 32'67 92:161 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:05:01.940351+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:35:40.654756+0000 2 0 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 4 0 0 0 0 2468 0 0 14 54 14 active+clean 2024-04-25T07:04:57.672944+0000 34'68 92:149 [1,2,0] 1 [1,2,0] 1 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:27:19.933857+0000 4 0 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 5 0 0 0 0 3085 0 0 14 54 14 active+clean 2024-04-25T07:04:58.918868+0000 36'68 92:174 [1,0,2] 1 [1,0,2] 1 36'68 2024-04-25T07:04:58.918670+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:10:05.200807+0000 5 0 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:05:14.353 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:05:14.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:05:14.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:05:14.518 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:05:14.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:05:14.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:05:14.680 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:05:14.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:05:14.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:05:14.833 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:05:14.845 INFO:tasks.workunit.client.0.smithi088.stdout:reading divergent objects 2024-04-25T07:05:14.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-04-25T07:05:14.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:05:15.234 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:05:15.234 INFO:tasks.workunit.client.0.smithi088.stdout:1.1f 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:57.662749+0000 34'68 92:187 [2,0,1] 2 [2,0,1] 2 34'68 2024-04-25T07:02:53.781717+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:59:13.669885+0000 3 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.1e 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:57.666128+0000 36'68 92:168 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:46.761850+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:02:11.818475+0000 2 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.1d 4 0 0 0 0 2468 0 0 15 54 15 active+clean 2024-04-25T07:04:58.399989+0000 34'69 92:182 [2,1,0] 2 [2,1,0] 2 34'69 2024-04-25T07:04:58.399782+0000 34'69 2024-04-25T07:02:55.069431+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:00:12.735488+0000 4 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.b 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:05:02.848314+0000 34'68 92:179 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:05:02.848111+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:11:50.597271+0000 3 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.a 3 0 0 0 0 1851 0 0 13 54 13 active+clean 2024-04-25T07:04:57.664849+0000 32'67 92:148 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:02:48.888019+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:22:31.276365+0000 3 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.9 5 0 0 0 0 3085 0 0 15 54 15 active+clean 2024-04-25T07:05:06.031659+0000 34'69 92:196 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:05:06.031443+0000 34'69 2024-04-25T07:05:06.031443+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:00:44.686488+0000 5 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.8 4 0 0 0 0 2526 0 0 15 54 15 active+clean 2024-04-25T07:04:57.661555+0000 34'69 92:193 [1,2,0] 1 [1,2,0] 1 34'69 2024-04-25T07:02:51.532335+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:05:04.836530+0000 32'67 92:180 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:05:04.836327+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:27.761509+0000 2 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 5 0 0 0 0 3085 0 0 16 54 16 active+clean 2024-04-25T07:04:57.673689+0000 81'70 92:176 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:03:01.636353+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:25:16.668383+0000 4 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 6 0 0 0 0 3702 0 0 15 54 15 active+clean 2024-04-25T07:04:57.638433+0000 34'69 92:192 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:53.093400+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:25:53.689638+0000 6 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:05:08.955855+0000 32'67 92:159 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:05:08.955656+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:13:07.564476+0000 2 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 4 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:04:57.664612+0000 36'68 92:170 [1,2,0] 1 [1,2,0] 1 36'68 2024-04-25T07:02:58.637424+0000 36'68 2024-04-25T07:02:58.637424+0000 0 0 queued for scrub 0 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 1 0 0 0 0 617 0 0 14 54 14 active+clean 2024-04-25T07:04:57.639260+0000 34'68 94:194 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:41.668117+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:00:18.042255+0000 0 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.c 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:05:06.925647+0000 34'68 92:181 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:05:06.925446+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:51:35.244414+0000 3 0 2024-04-25T07:05:15.235 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 2 0 0 0 0 1234 0 0 15 54 15 active+clean 2024-04-25T07:04:57.661863+0000 34'69 92:199 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:02:54.006669+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:29:26.471755+0000 2 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.1c 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:05:03.881404+0000 32'67 92:169 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:05:03.881196+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:06:18.017832+0000 2 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:04:57.667762+0000 34'68 92:153 [1,0,2] 1 [1,0,2] 1 32'67 2024-04-25T07:02:40.631601+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:55:23.652870+0000 2 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.d 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:04:57.641049+0000 34'69 92:193 [2,1,0] 2 [2,1,0] 2 34'69 2024-04-25T07:02:51.065172+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:00:25.428703+0000 3 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.e 4 0 0 0 0 2818 0 0 16 54 16 active+clean 2024-04-25T07:04:59.919507+0000 73'70 92:193 [1,2,0] 1 [1,2,0] 1 73'70 2024-04-25T07:04:59.919281+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:47:53.870417+0000 4 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.f 4 0 0 0 0 2468 0 0 13 54 13 active+clean 2024-04-25T07:04:57.639346+0000 32'67 92:181 [2,0,1] 2 [2,0,1] 2 32'67 2024-04-25T07:02:46.980509+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:16:13.802342+0000 4 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.10 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:05:03.148568+0000 34'68 94:183 [0,2,1] 0 [0,2,1] 0 34'68 2024-04-25T07:05:03.148374+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:48:10.936244+0000 3 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.11 3 0 0 0 0 1851 0 0 13 54 13 active+clean+inconsistent 2024-04-25T07:05:02.196729+0000 32'67 94:170 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:05:02.196381+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:23:56.305176+0000 3 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.12 1 0 0 0 0 617 0 0 13 54 13 active+clean 2024-04-25T07:04:57.615521+0000 32'67 94:181 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:02:49.759395+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:39:58.170882+0000 1 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.13 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:05:07.912992+0000 34'69 92:190 [1,0,2] 1 [1,0,2] 1 34'69 2024-04-25T07:05:07.912761+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:34:36.907655+0000 3 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.14 3 0 0 0 0 1851 0 0 15 54 15 active+clean 2024-04-25T07:05:04.384861+0000 34'69 92:129 [2,0,1] 2 [2,0,1] 2 34'69 2024-04-25T07:05:04.384533+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:21:03.674049+0000 3 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.15 3 0 0 0 0 1909 0 0 14 54 14 active+clean 2024-04-25T07:05:06.332237+0000 34'68 92:116 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:50.078073+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 queued for scrub 0 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.16 4 0 0 0 0 2468 0 0 14 54 14 active+clean 2024-04-25T07:04:57.668312+0000 34'68 92:168 [1,0,2] 1 [1,0,2] 1 34'68 2024-04-25T07:02:53.871689+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:44:07.604489+0000 4 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.17 2 0 0 0 0 1292 0 0 13 54 13 peering 2024-04-25T07:03:20.498757+0000 32'67 94:95 [0,1,2] 0 [0,1,2] 0 32'67 2024-04-25T07:05:09.961881+0000 0'0 2024-04-25T07:02:16.902637+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:05:15.236 INFO:tasks.workunit.client.0.smithi088.stdout:1.18 3 0 0 0 0 1851 0 0 14 54 14 active+clean 2024-04-25T07:05:10.937769+0000 35'68 92:200 [1,2,0] 1 [1,2,0] 1 35'68 2024-04-25T07:05:10.937567+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:25:21.543892+0000 3 0 2024-04-25T07:05:15.237 INFO:tasks.workunit.client.0.smithi088.stdout:1.19 2 0 0 0 0 1234 0 0 13 54 13 active+clean 2024-04-25T07:05:01.940676+0000 32'67 92:161 [1,2,0] 1 [1,2,0] 1 32'67 2024-04-25T07:05:01.940351+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:35:40.654756+0000 2 0 2024-04-25T07:05:15.237 INFO:tasks.workunit.client.0.smithi088.stdout:1.1a 4 0 0 0 0 2468 0 0 14 54 14 active+clean 2024-04-25T07:04:57.672944+0000 34'68 92:149 [1,2,0] 1 [1,2,0] 1 34'68 2024-04-25T07:02:52.859592+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:27:19.933857+0000 4 0 2024-04-25T07:05:15.237 INFO:tasks.workunit.client.0.smithi088.stdout:1.1b 5 0 0 0 0 3085 0 0 14 54 14 active+clean 2024-04-25T07:04:58.918868+0000 36'68 92:174 [1,0,2] 1 [1,0,2] 1 36'68 2024-04-25T07:04:58.918670+0000 0'0 2024-04-25T07:02:16.902637+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:10:05.200807+0000 5 0 2024-04-25T07:05:15.237 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:05:15.237 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:05:15.237 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:05:15.250 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-04-25T07:05:15.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-04-25T07:05:15.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:05:15.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:05:15.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:05:15.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:05:15.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:05:15.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:05:15.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:05:15.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:05:15.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-04-25T07:05:15.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:05:15.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-04-25T07:05:15.450 INFO:tasks.workunit.client.0.smithi088.stderr:error getting test/newobject_1: (2) No such file or directory 2024-04-25T07:05:15.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:05:15.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-04-25T07:05:15.484 INFO:tasks.workunit.client.0.smithi088.stderr:error getting test/newobject_2: (2) No such file or directory 2024-04-25T07:05:15.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-04-25T07:05:15.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:05:15.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:05:15.507 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 41'70 (31'4) delete 1:70b3bf12:::existing_4:head by client.4630.0:1 2024-04-25T07:02:54.763006+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:05:15.507 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 20 _merge_object_divergent_entries: keeping 41'70 (31'4) delete 1:70b3bf12:::existing_4:head by client.4630.0:1 2024-04-25T07:02:54.763006+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:05:15.507 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:05:15.507 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 41'70 last_divergent_update: 41'70 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: more recent entry found: 73'70 (31'4) modify 1:70b3bf12:::existing_4:head by client.4750.0:1 2024-04-25T07:03:59.376586+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 41'71 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-04-25T07:02:54.764286+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 20 _merge_object_divergent_entries: keeping 41'71 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-04-25T07:02:54.764286+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 41'71 last_divergent_update: 41'71 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 41'70 (31'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-04-25T07:02:54.766371+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 20 _merge_object_divergent_entries: keeping 41'70 (31'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-04-25T07:02:54.766371+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 41'70 last_divergent_update: 41'70 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:05:15.508 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 41'70 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 41'70 (31'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-04-25T07:02:54.766371+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'54 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 41'69 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4628.0:1 2024-04-25T07:02:54.708557+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 20 _merge_object_divergent_entries: keeping 41'69 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4628.0:1 2024-04-25T07:02:54.708557+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 41'69 last_divergent_update: 41'69 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.430+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 41'70 (31'2) modify 1:97db2a0f:::existing_2:head by client.4629.0:1 2024-04-25T07:02:54.762202+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 20 _merge_object_divergent_entries: keeping 41'70 (31'2) modify 1:97db2a0f:::existing_2:head by client.4629.0:1 2024-04-25T07:02:54.762202+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 41'70 last_divergent_update: 41'70 2024-04-25T07:05:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:05:15.510 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:05:15.510 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 41'70 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 41'70 (31'2) modify 1:97db2a0f:::existing_2:head by client.4629.0:1 2024-04-25T07:02:54.762202+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.442+0000 7f5d6b5e3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'54 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 41'68 (31'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:02:54.764566+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 20 _merge_object_divergent_entries: keeping 41'68 (31'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:02:54.764566+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 41'68 last_divergent_update: 41'68 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:05:15.519 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 41'68 2024-04-25T07:05:15.520 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 41'68 (31'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:02:54.764566+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:05:15.520 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:05:15.520 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:04:46.858+0000 7f5d6c5e5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'54 2024-04-25T07:05:15.520 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 41'68 (31'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:02:54.764566+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:05:15.520 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 20 _merge_object_divergent_entries: keeping 41'68 (31'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:02:54.764566+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:05:15.520 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:05:15.520 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 41'68 last_divergent_update: 41'68 2024-04-25T07:05:15.521 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:05:15.521 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:05:15.521 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 41'68 2024-04-25T07:05:15.521 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 41'68 (31'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-04-25T07:02:54.764566+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:05:15.521 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:05:15.521 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:04:46.430+0000 7f53a1b67640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'54 2024-04-25T07:05:15.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:05:15.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:05:15.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:05:15.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:05:15.567 INFO:tasks.workunit.client.0.smithi088.stdout:success 2024-04-25T07:05:15.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-04-25T07:05:15.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.26017.out 2024-04-25T07:05:15.568 INFO:tasks.workunit.client.0.smithi088.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.26017.out': No such file or directory 2024-04-25T07:05:15.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-04-25T07:05:15.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:05:15.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:05:16.240 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T07:05:16.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-04-25T07:05:16.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:16.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:16.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:16.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:16.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:16.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:16.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:05:16.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:05:16.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:05:16.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:05:16.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:16.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:16.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:16.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:16.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:16.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:16.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:05:16.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:05:16.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:05:16.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:05:16.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:05:16.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:05:16.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:16.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:05:16.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:05:16.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:16.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:05:16.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:05:16.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:05:16.621 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:05:16.621 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:16.621 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:16.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:16.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:05:16.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:05:16.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:05:16.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:05:16.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:05:16.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:05:16.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:16.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:05:16.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:05:16.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:16.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:05:16.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:05:16.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:05:16.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:05:16.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:16.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:16.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T07:05:16.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:05:16.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:05:16.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:05:16.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:05:16.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:16.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:16.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26017 2024-04-25T07:05:16.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:05:16.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:05:16.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:05:16.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:05:16.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-04-25T07:05:16.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-04-25T07:05:16.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-04-25T07:05:16.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-04-25T07:05:16.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-04-25T07:05:16.636 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-04-25T07:05:16.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:05:16.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:05:17.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:05:17.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:17.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:17.086 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:17.086 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:17.086 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:17.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:05:17.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:05:17.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:05:17.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:05:17.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:05:17.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:05:17.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:05:17.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:05:17.290 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:05:17.290 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:05:17.290 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:05:17.291 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:17.291 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:17.291 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:17.291 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-mon.a.asok 2024-04-25T07:05:17.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:05:17.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26017/ceph-mon.a.asok config get fsid 2024-04-25T07:05:17.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:05:17.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:05:17.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:05:17.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:05:17.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:05:17.409 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:05:17.409 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:05:17.409 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:05:17.410 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:17.410 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:17.410 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:17.410 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-mon.a.asok 2024-04-25T07:05:17.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:05:17.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26017/ceph-mon.a.asok config get mon_host 2024-04-25T07:05:17.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-04-25T07:05:17.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:05:17.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:05:17.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:05:17.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:05:17.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:05:17.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:05:18.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:05:18.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:18.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:18.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:18.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:18.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:18.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:05:18.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:05:18.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:05:18.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:05:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:05:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:05:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:05:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:05:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:05:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:05:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:05:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:05:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:05:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:05:18.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:05:18.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e25fde4-eacc-4fda-876f-6ce01e9c6922 2024-04-25T07:05:18.140 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 9e25fde4-eacc-4fda-876f-6ce01e9c6922 2024-04-25T07:05:18.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9e25fde4-eacc-4fda-876f-6ce01e9c6922' 2024-04-25T07:05:18.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:05:18.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuACpm4dDnCRAA0xiBHx41Wh6d70pk2ftlAA== 2024-04-25T07:05:18.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuACpm4dDnCRAA0xiBHx41Wh6d70pk2ftlAA=="}' 2024-04-25T07:05:18.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e25fde4-eacc-4fda-876f-6ce01e9c6922 -i td/divergent-priors/0/new.json 2024-04-25T07:05:18.466 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:05:18.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:05:18.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCuACpm4dDnCRAA0xiBHx41Wh6d70pk2ftlAA== --osd-uuid 9e25fde4-eacc-4fda-876f-6ce01e9c6922 2024-04-25T07:05:18.525 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:18.530+0000 7fc08a9927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:18.534 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:18.538+0000 7fc08a9927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:18.535 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:18.538+0000 7fc08a9927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:18.536 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:18.538+0000 7fc08a9927c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:05:18.536 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:18.538+0000 7fc08a9927c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:05:18.536 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:18.538+0000 7fc08a9927c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:05:18.574 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:18.578+0000 7fc08a9927c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:05:24.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:05:24.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:05:24.394 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:05:24.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:05:24.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:05:25.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:05:25.026 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:05:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:05:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:05:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:05:25.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:05:25.066 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:25.069+0000 7f79ba3977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:25.068 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:25.073+0000 7f79ba3977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:25.069 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:25.073+0000 7f79ba3977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:25.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:05:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:05:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:05:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:05:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:05:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:05:25.444 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:05:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:25.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:25.610 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:25.613+0000 7f79ba3977c0 -1 Falling back to public interface 2024-04-25T07:05:25.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:26.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:26.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:26.922 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:05:26.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:05:26.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:26.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:27.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:28.147 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:28.149+0000 7f79ba3977c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:05:28.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:28.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:28.407 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:05:28.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:05:28.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:28.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:28.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:29.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:29.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:29.843 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:05:29.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:05:29.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:29.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:30.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:31.289 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:05:31.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:05:31.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:31.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:31.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:32.853 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:05:32.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:05:32.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:32.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:33.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:34.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:34.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:34.290 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:05:34.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:05:34.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:34.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:34.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:35.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:35.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:35.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:05:35.881 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:05:35.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:35.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:36.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:37.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:37.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:37.308 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:05:37.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:05:37.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:37.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:37.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:38.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:38.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:38.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:05:38.845 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:05:38.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:38.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:39.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:40.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:40.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:40.375 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:05:40.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:05:40.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:40.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:41.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:42.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:42.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:42.017 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:05:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:05:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:42.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:43.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:43.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:43.458 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:05:43.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:05:43.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:43.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:44.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:45.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:45.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:45.250 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:05:45.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:05:45.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:45.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:46.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:47.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:47.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:47.171 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:05:47.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:05:47.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:47.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:48.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:49.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:49.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:49.270 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:05:49.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:05:49.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:49.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:49.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:50.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:50.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:50.897 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:05:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:05:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:51.342 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4220446258,v1:127.0.0.1:6803/4220446258] [v2:127.0.0.1:6804/4220446258,v1:127.0.0.1:6805/4220446258] exists,up 9e25fde4-eacc-4fda-876f-6ce01e9c6922 2024-04-25T07:05:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:05:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:05:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:05:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:05:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-04-25T07:05:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:05:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:05:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:05:51.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:05:51.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:51.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:05:51.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:05:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:05:51.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:05:51.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b02c5640-7b84-4669-94ce-bea6cb51d86c 2024-04-25T07:05:51.349 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 b02c5640-7b84-4669-94ce-bea6cb51d86c 2024-04-25T07:05:51.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b02c5640-7b84-4669-94ce-bea6cb51d86c' 2024-04-25T07:05:51.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:05:51.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPACpme2ZrFhAAmdkAOdfgB7qc0IFY8aMdiA== 2024-04-25T07:05:51.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPACpme2ZrFhAAmdkAOdfgB7qc0IFY8aMdiA=="}' 2024-04-25T07:05:51.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b02c5640-7b84-4669-94ce-bea6cb51d86c -i td/divergent-priors/1/new.json 2024-04-25T07:05:51.911 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:05:51.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:05:51.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPACpme2ZrFhAAmdkAOdfgB7qc0IFY8aMdiA== --osd-uuid b02c5640-7b84-4669-94ce-bea6cb51d86c 2024-04-25T07:05:51.972 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:51.973+0000 7f3188a457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:51.973 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:51.977+0000 7f3188a457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:51.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:51.977+0000 7f3188a457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:51.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:51.977+0000 7f3188a457c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:05:51.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:51.977+0000 7f3188a457c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:05:51.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:51.977+0000 7f3188a457c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:05:52.013 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:52.017+0000 7f3188a457c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:05:57.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:05:57.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:05:57.925 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:05:57.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:05:57.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:05:58.503 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:05:58.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:05:58.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:05:58.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:05:58.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:05:58.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:05:58.543 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:58.545+0000 7f1d6a10b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:58.560 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:58.565+0000 7f1d6a10b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:58.562 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:58.565+0000 7f1d6a10b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:58.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:05:58.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:05:58.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:05:58.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:05:58.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:05:58.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:58.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:05:58.948 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:05:58.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:05:58.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:59.122 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:05:59.125+0000 7f1d6a10b7c0 -1 Falling back to public interface 2024-04-25T07:05:59.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:00.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:00.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:00.411 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:06:00.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:06:00.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:00.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:00.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:01.696 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:01.700+0000 7f1d6a10b7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:06:01.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:01.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:01.884 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:06:01.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:06:01.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:01.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:02.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:03.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:03.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:03.495 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:06:03.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:06:03.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:03.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:04.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:05.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:05.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:05.018 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:06:05.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:06:05.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:05.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:05.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:06.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:06.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:06.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:06:06.497 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:06:06.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:06.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:06.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:07.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:07.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:07.930 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:06:07.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:06:07.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:07.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:08.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:09.396 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:06:09.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:06:09.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:09.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:09.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:10.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:10.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:10.898 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:06:10.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:06:10.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:10.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:11.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:12.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:12.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:12.480 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:06:12.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:06:12.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:12.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:12.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:13.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:13.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:13.920 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:06:13.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:06:13.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:13.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:14.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:15.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:15.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:15.536 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:06:15.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:06:15.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:15.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:16.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:17.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:17.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:17.574 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:06:17.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:06:17.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:17.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:18.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:19.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:19.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:19.478 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:06:19.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:06:19.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:19.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:20.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:21.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:21.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:21.418 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:06:21.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:06:21.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:21.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:22.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:23.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:23.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:23.317 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:06:23.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:06:23.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:23.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:24.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:25.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:25.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:25.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:06:25.099 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:06:25.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:25.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:25.512 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2807942619,v1:127.0.0.1:6811/2807942619] [v2:127.0.0.1:6812/2807942619,v1:127.0.0.1:6813/2807942619] exists,up b02c5640-7b84-4669-94ce-bea6cb51d86c 2024-04-25T07:06:25.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:06:25.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:06:25.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:06:25.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:06:25.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-04-25T07:06:25.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:06:25.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:06:25.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:06:25.516 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:06:25.516 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:06:25.516 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:06:25.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:06:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:06:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:06:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:06:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:06:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:06:25.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:06:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a5fd889-264a-480b-8a6e-79c26bb25857 2024-04-25T07:06:25.519 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 5a5fd889-264a-480b-8a6e-79c26bb25857 2024-04-25T07:06:25.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5a5fd889-264a-480b-8a6e-79c26bb25857' 2024-04-25T07:06:25.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:06:25.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxACpmNu3bIBAAhY5QrLhg5lzD8H89F+sRJQ== 2024-04-25T07:06:25.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxACpmNu3bIBAAhY5QrLhg5lzD8H89F+sRJQ=="}' 2024-04-25T07:06:25.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a5fd889-264a-480b-8a6e-79c26bb25857 -i td/divergent-priors/2/new.json 2024-04-25T07:06:26.080 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:06:26.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:06:26.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxACpmNu3bIBAAhY5QrLhg5lzD8H89F+sRJQ== --osd-uuid 5a5fd889-264a-480b-8a6e-79c26bb25857 2024-04-25T07:06:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:26.136+0000 7effdbeb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:26.153 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:26.156+0000 7effdbeb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:26.154 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:26.156+0000 7effdbeb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:26.154 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:26.156+0000 7effdbeb27c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:06:26.155 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:26.156+0000 7effdbeb27c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:06:26.155 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:26.156+0000 7effdbeb27c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:06:26.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:26.240+0000 7effdbeb27c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:06:31.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:06:31.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:06:31.804 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:06:31.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:06:31.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:06:32.378 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:06:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:06:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:06:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:06:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:06:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:06:32.418 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:32.420+0000 7f60710917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:32.514 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:32.516+0000 7f60710917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:32.515 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:32.516+0000 7f60710917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:06:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:06:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:06:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:06:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:06:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:32.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:06:32.854 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:06:32.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:32.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:33.087 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:33.088+0000 7f60710917c0 -1 Falling back to public interface 2024-04-25T07:06:33.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:34.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:34.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:34.298 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:06:34.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:06:34.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:34.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:34.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:35.632 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:06:35.632+0000 7f60710917c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:06:35.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:35.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:35.733 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:06:35.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:06:35.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:35.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:37.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:37.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:37.324 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:06:37.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:06:37.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:37.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:37.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:38.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:38.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:38.784 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:06:38.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:06:38.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:38.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:39.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:40.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:40.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:40.230 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:06:40.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:06:40.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:40.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:40.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:41.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:41.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:41.693 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:06:41.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:06:41.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:41.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:42.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:43.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:43.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:43.140 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:06:43.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:06:43.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:43.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:43.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:44.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:44.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:44.583 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:06:44.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:06:44.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:44.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:45.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:46.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:46.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:46.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:06:46.110 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:06:46.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:46.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:46.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:47.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:47.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:47.569 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:06:47.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:06:47.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:47.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:48.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:49.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:49.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:49.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:06:49.379 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:06:49.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:49.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:50.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:51.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:51.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:51.320 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:06:51.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:06:51.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:51.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:52.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:53.314 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:06:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:06:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:53.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:54.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:54.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:54.993 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:06:54.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:06:54.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:54.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:55.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:56.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:56.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:56.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:06:56.733 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:06:56.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:56.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:57.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:58.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:58.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:58.178 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:06:58.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:06:58.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:58.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:58.641 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3734996299,v1:127.0.0.1:6819/3734996299] [v2:127.0.0.1:6820/3734996299,v1:127.0.0.1:6821/3734996299] exists,up 5a5fd889-264a-480b-8a6e-79c26bb25857 2024-04-25T07:06:58.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:06:58.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:06:58.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:06:58.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-04-25T07:06:59.551 INFO:tasks.workunit.client.0.smithi088.stderr:noout is set 2024-04-25T07:06:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-04-25T07:07:00.329 INFO:tasks.workunit.client.0.smithi088.stderr:noin is set 2024-04-25T07:07:00.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-04-25T07:07:01.131 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T07:07:01.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-04-25T07:07:01.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=test 2024-04-25T07:07:01.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-04-25T07:07:01.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-04-25T07:07:01.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-04-25T07:07:01.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-25T07:07:01.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-04-25T07:07:01.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T07:07:02.590 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T07:07:02.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:07:03.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-04-25T07:07:03.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-04-25T07:07:04.273 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 allow_ec_overwrites to true 2024-04-25T07:07:04.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-04-25T07:07:04.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:07:04.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:07:04.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:07:04.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:07:04.296 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:07:04.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:07:04.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:07:04.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:07:04.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:07:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:07:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:07:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:07:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:07:04.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:07:04.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:07:04.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:07:04.867 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:07:04.867 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:07:04.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:07:04.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:04.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:07:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-04-25T07:07:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-04-25T07:07:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-04-25T07:07:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:05.036 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:07:05.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T07:07:05.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T07:07:05.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672975' 2024-04-25T07:07:05.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:05.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:07:05.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-04-25T07:07:05.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-04-25T07:07:05.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672975 2-64424509448' 2024-04-25T07:07:05.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:05.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-04-25T07:07:05.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:05.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:07:05.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-04-25T07:07:05.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:05.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-04-25T07:07:05.354 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836503 2024-04-25T07:07:05.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-25T07:07:05.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:05.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836503 2024-04-25T07:07:05.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:07:06.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:07:06.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:07.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836503 2024-04-25T07:07:07.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:07.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T07:07:07.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:07.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:07:07.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T07:07:07.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:07.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T07:07:07.207 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672975 2024-04-25T07:07:07.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T07:07:07.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:07:07.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672975 2024-04-25T07:07:07.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:07.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-04-25T07:07:07.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:07.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:07:07.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-04-25T07:07:07.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:07.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-04-25T07:07:07.615 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509448 2024-04-25T07:07:07.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-25T07:07:07.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:07:08.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509448 -lt 64424509448 2024-04-25T07:07:08.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:07:08.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:08.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:08.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:07:08.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:07:08.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:07:08.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:07:08.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:07:08.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:07:08.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:07:08.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:07:09.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:07:09.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:07:09.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:09.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:07:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:07:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:07:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-04-25T07:07:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-04-25T07:07:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:07:09.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:07:10.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:07:10.081 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:07:10.081 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:07:10.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:07:10.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:10.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:07:10.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-04-25T07:07:10.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-04-25T07:07:10.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-04-25T07:07:10.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:10.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:07:10.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-04-25T07:07:10.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-04-25T07:07:10.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672979' 2024-04-25T07:07:10.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:10.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:07:10.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-04-25T07:07:10.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-04-25T07:07:10.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509451' 2024-04-25T07:07:10.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:10.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-04-25T07:07:10.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:10.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:07:10.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-04-25T07:07:10.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:10.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-04-25T07:07:10.556 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836506 2024-04-25T07:07:10.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-04-25T07:07:10.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:10.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836506 2024-04-25T07:07:10.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:10.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-04-25T07:07:10.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:10.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:07:10.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-04-25T07:07:10.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:10.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-04-25T07:07:10.974 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672979 2024-04-25T07:07:10.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-04-25T07:07:10.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:07:11.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672979 2024-04-25T07:07:11.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:11.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-04-25T07:07:11.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:11.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:07:11.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-04-25T07:07:11.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:11.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-04-25T07:07:11.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-25T07:07:11.414 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509451 2024-04-25T07:07:11.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:07:11.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-25T07:07:11.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-04-25T07:07:11.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:07:11.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:07:11.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:07:11.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:07:11.902 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:07:11.902 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:07:11.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:07:11.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:07:11.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:07:12.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:07:12.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:07:12.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:07:12.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:07:12.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:07:12.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:07:12.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:07:12.485 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:07:12.486 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:07:12.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:07:12.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:12.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:07:12.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836509 2024-04-25T07:07:12.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836509 2024-04-25T07:07:12.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509' 2024-04-25T07:07:12.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:12.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:07:12.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-04-25T07:07:12.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-04-25T07:07:12.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-04-25T07:07:12.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:12.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:07:12.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509453 2024-04-25T07:07:12.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509453 2024-04-25T07:07:12.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509453' 2024-04-25T07:07:12.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:12.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836509 2024-04-25T07:07:12.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:13.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:07:13.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-04-25T07:07:13.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:13.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836509 2024-04-25T07:07:13.002 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836509 2024-04-25T07:07:13.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-04-25T07:07:13.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:13.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836509 2024-04-25T07:07:13.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:13.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-04-25T07:07:13.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:13.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:07:13.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-04-25T07:07:13.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:13.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-04-25T07:07:13.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-04-25T07:07:13.477 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672981 2024-04-25T07:07:13.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:07:13.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672981 2024-04-25T07:07:13.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:07:14.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:07:14.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:07:15.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672981 -lt 42949672981 2024-04-25T07:07:15.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:15.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509453 2024-04-25T07:07:15.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:15.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:07:15.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-04-25T07:07:15.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:15.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509453 2024-04-25T07:07:15.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-04-25T07:07:15.348 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509453 2024-04-25T07:07:15.348 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:07:15.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509453 2024-04-25T07:07:15.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:07:15.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:15.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:16.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:07:16.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:07:16.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:07:16.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:07:16.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:07:16.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:07:16.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:07:16.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:07:16.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:07:16.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:07:16.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:16.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:17.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:07:17.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:07:17.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:07:17.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-04-25T07:07:17.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-04-25T07:07:17.713 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:07:17.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-04-25T07:07:17.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-04-25T07:07:17.730 INFO:tasks.workunit.client.0.smithi088.stdout:primary and soon to be divergent is 1 2024-04-25T07:07:17.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:07:18.134 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:07:18.134 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:07:03.482055+0000 0'0 29:17 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:09:56.922032+0000 0 0 2024-04-25T07:07:18.134 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:07:18.134 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:07:18.135 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:07:18.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-04-25T07:07:18.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:07:18.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-04-25T07:07:18.152 INFO:tasks.workunit.client.0.smithi088.stdout:writing initial objects 2024-04-25T07:07:18.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-04-25T07:07:18.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:18.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-04-25T07:07:18.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:18.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-04-25T07:07:18.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:18.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-04-25T07:07:18.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:18.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-04-25T07:07:18.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:18.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-04-25T07:07:19.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:19.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-04-25T07:07:19.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:19.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-04-25T07:07:19.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-04-25T07:07:19.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:19.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-04-25T07:07:19.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:19.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-04-25T07:07:20.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:20.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-04-25T07:07:20.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:20.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-04-25T07:07:20.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:20.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-04-25T07:07:20.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:20.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-04-25T07:07:20.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:20.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-04-25T07:07:21.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:21.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-04-25T07:07:21.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:21.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-04-25T07:07:21.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:21.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-04-25T07:07:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:21.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-04-25T07:07:21.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:21.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-04-25T07:07:22.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:22.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-04-25T07:07:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:22.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-04-25T07:07:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-04-25T07:07:22.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:22.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-04-25T07:07:22.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:22.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-04-25T07:07:23.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:23.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-04-25T07:07:23.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:23.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-04-25T07:07:23.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:23.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-04-25T07:07:23.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:23.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-04-25T07:07:23.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:23.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-04-25T07:07:24.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-04-25T07:07:24.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-04-25T07:07:24.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-04-25T07:07:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-04-25T07:07:24.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:24.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-04-25T07:07:25.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-04-25T07:07:25.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-04-25T07:07:25.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-04-25T07:07:25.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-04-25T07:07:25.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:25.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-04-25T07:07:26.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-04-25T07:07:26.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-04-25T07:07:26.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-04-25T07:07:26.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-04-25T07:07:26.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:26.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-04-25T07:07:27.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-04-25T07:07:27.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-04-25T07:07:27.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-04-25T07:07:27.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-04-25T07:07:27.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-04-25T07:07:27.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:27.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-04-25T07:07:28.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-04-25T07:07:28.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-04-25T07:07:28.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-04-25T07:07:28.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:28.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-04-25T07:07:29.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-04-25T07:07:29.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-04-25T07:07:29.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-04-25T07:07:29.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-04-25T07:07:29.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-04-25T07:07:29.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:29.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-04-25T07:07:30.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-04-25T07:07:30.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-04-25T07:07:30.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-04-25T07:07:30.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-04-25T07:07:30.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:30.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-04-25T07:07:31.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-04-25T07:07:31.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-04-25T07:07:31.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-04-25T07:07:31.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-04-25T07:07:31.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:31.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-04-25T07:07:32.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-04-25T07:07:32.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-04-25T07:07:32.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-04-25T07:07:32.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-04-25T07:07:32.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:32.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-04-25T07:07:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-04-25T07:07:33.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-04-25T07:07:33.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-04-25T07:07:33.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-04-25T07:07:33.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:33.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-04-25T07:07:34.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-04-25T07:07:34.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-04-25T07:07:34.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-04-25T07:07:34.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-04-25T07:07:34.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:34.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-04-25T07:07:35.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-04-25T07:07:35.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-04-25T07:07:35.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-04-25T07:07:35.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-04-25T07:07:35.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:35.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-04-25T07:07:36.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-04-25T07:07:36.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-04-25T07:07:36.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-04-25T07:07:36.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-04-25T07:07:36.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-04-25T07:07:36.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:36.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-04-25T07:07:37.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-04-25T07:07:37.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-04-25T07:07:37.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:07:37.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-04-25T07:07:37.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:07:37.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-04-25T07:07:37.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:07:37.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:07:37.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:07:37.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:07:37.760 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:07:37.760 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:07:37.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:07:37.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:07:37.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:07:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:07:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:07:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:07:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:07:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:07:37.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:07:38.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:07:38.284 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:07:38.284 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:07:38.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:07:38.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:38.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:07:38.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-04-25T07:07:38.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-04-25T07:07:38.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-04-25T07:07:38.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:38.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:07:38.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672988 2024-04-25T07:07:38.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672988 2024-04-25T07:07:38.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672988' 2024-04-25T07:07:38.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:07:38.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:07:38.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509461 2024-04-25T07:07:38.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509461 2024-04-25T07:07:38.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672988 2-64424509461' 2024-04-25T07:07:38.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:38.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-04-25T07:07:38.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:38.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:07:38.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-04-25T07:07:38.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:38.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-04-25T07:07:38.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-04-25T07:07:38.797 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836516 2024-04-25T07:07:38.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:07:39.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836516 -lt 21474836516 2024-04-25T07:07:39.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:39.243 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672988 2024-04-25T07:07:39.243 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:39.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:07:39.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672988 2024-04-25T07:07:39.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:39.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672988 2024-04-25T07:07:39.247 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672988 2024-04-25T07:07:39.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-04-25T07:07:39.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:07:39.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672988 -lt 42949672988 2024-04-25T07:07:39.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:07:39.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509461 2024-04-25T07:07:39.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:07:39.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:07:39.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509461 2024-04-25T07:07:39.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:07:39.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509461 2024-04-25T07:07:39.675 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509461 2024-04-25T07:07:39.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-04-25T07:07:39.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:07:40.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509459 -lt 64424509461 2024-04-25T07:07:40.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:07:41.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:07:41.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:07:41.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509461 -lt 64424509461 2024-04-25T07:07:41.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:07:41.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:41.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:42.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:07:42.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:07:42.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:07:42.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:07:42.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:07:42.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:07:42.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:07:42.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:07:42.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:07:42.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:07:42.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:07:42.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:07:43.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:07:43.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:07:43.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:07:43.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-04-25T07:07:43.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:07:43.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:07:43.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:07:43.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:07:43.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-04-25T07:07:43.477 INFO:tasks.workunit.client.0.smithi088.stdout:blackholing osds 0 2 2024-04-25T07:07:43.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-04-25T07:07:43.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:07:43.876 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:07:43.877 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:07:03.482055+0000 29'100 29:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:09:56.922032+0000 0 0 2024-04-25T07:07:43.877 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:07:43.877 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:07:43.877 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:07:43.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:07:43.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-04-25T07:07:43.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:07:43.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:07:43.893 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:07:43.893 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:43.894 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:07:43.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.0.asok 2024-04-25T07:07:43.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:07:43.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:07:43.989 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:07:43.989 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:07:43.989 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:07:44.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:07:44.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-04-25T07:07:44.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:07:44.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:07:44.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:07:44.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:44.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:07:44.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.2.asok 2024-04-25T07:07:44.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:07:44.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:07:44.111 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:07:44.111 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:07:44.111 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:07:44.123 INFO:tasks.workunit.client.0.smithi088.stdout:writing divergent object 2024-04-25T07:07:44.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-04-25T07:07:44.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-04-25T07:07:44.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-04-25T07:07:45.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-04-25T07:07:45.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-04-25T07:07:45.411 INFO:tasks.workunit.client.0.smithi088.stdout:created pool test snap snap1 2024-04-25T07:07:45.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-04-25T07:07:45.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-04-25T07:07:45.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-04-25T07:07:55.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-04-25T07:07:55.421 INFO:tasks.workunit.client.0.smithi088.stderr:rados: no process found 2024-04-25T07:07:55.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-04-25T07:07:55.422 INFO:tasks.workunit.client.0.smithi088.stdout:killing all the osds 2024-04-25T07:07:55.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:07:55.829 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:07:55.829 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-04-25T07:07:03.482055+0000 30'104 30:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:20:19.627848+0000 0 0 2024-04-25T07:07:55.829 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:07:55.829 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:07:55.830 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:07:55.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:07:55.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:07:55.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:07:55.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:07:55.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:07:55.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:07:55.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:07:55.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:07:55.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-04-25T07:07:56.667 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T07:07:56.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:07:56.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-04-25T07:07:57.218 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T07:07:57.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:07:57.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-04-25T07:07:57.752 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already down. 2024-04-25T07:07:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:07:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-04-25T07:07:58.328 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T07:07:58.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:07:58.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-04-25T07:07:58.871 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already out. 2024-04-25T07:07:58.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-04-25T07:07:58.887 INFO:tasks.workunit.client.0.smithi088.stdout:bringing up non_divergent 0 2 2024-04-25T07:07:58.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:07:59.283 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:07:59.283 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-04-25T07:07:03.482055+0000 30'104 30:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:20:19.627848+0000 0 0 2024-04-25T07:07:59.283 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:07:59.283 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:07:59.284 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:07:59.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:07:59.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-04-25T07:07:59.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:07:59.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:07:59.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:07:59.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:07:59.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:07:59.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:07:59.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:07:59.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:07:59.303 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:07:59.303 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:59.303 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:07:59.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:07:59.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:07:59.305 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:07:59.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:07:59.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:07:59.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:07:59.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:07:59.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:07:59.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:07:59.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:07:59.343 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:07:59.342+0000 7f843351e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:59.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:07:59.346+0000 7f843351e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:59.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:07:59.346+0000 7f843351e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:59.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:07:59.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:59.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:07:59.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:59.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:59.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:59.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:59.709 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:07:59.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:59.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:59.906 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:07:59.906+0000 7f843351e7c0 -1 Falling back to public interface 2024-04-25T07:08:00.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:01.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:01.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:01.175 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:08:01.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:08:01.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:01.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:01.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:02.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:02.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:02.601 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:08:02.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:08:02.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:02.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:02.740 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:02.738+0000 7f843351e7c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:08:03.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:04.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:04.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:04.080 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:08:04.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:08:04.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:04.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:04.557 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/2206864998,v1:127.0.0.1:6803/2206864998] [v2:127.0.0.1:6804/2206864998,v1:127.0.0.1:6805/2206864998] exists,up 9e25fde4-eacc-4fda-876f-6ce01e9c6922 2024-04-25T07:08:04.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:08:04.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:04.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:04.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:04.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:04.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:04.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:08:04.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:08:04.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:08:04.562 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:08:04.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:08:04.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:04.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:08:04.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:08:04.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:08:04.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:08:04.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:08:04.603 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:04.602+0000 7f8271ef17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:04.630 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:04.630+0000 7f8271ef17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:04.632 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:04.630+0000 7f8271ef17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:05.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:08:05.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:05.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:08:05.018 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:08:05.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:05.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:05.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:05.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:05.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:05.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:05.206 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:05.206+0000 7f8271ef17c0 -1 Falling back to public interface 2024-04-25T07:08:05.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:06.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:06.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:06.469 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:08:06.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:08:06.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:06.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:06.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:07.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:07.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:07.919 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:08:07.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:08:07.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:07.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:08.138 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:08.138+0000 7f8271ef17c0 -1 osd.2 29 log_to_monitors true 2024-04-25T07:08:08.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:09.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:09.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:09.413 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:08:09.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:08:09.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:09.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:09.860 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/2144286097,v1:127.0.0.1:6811/2144286097] [v2:127.0.0.1:6812/2144286097,v1:127.0.0.1:6813/2144286097] exists,up 5a5fd889-264a-480b-8a6e-79c26bb25857 2024-04-25T07:08:09.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:09.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:09.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:09.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:08:09.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-04-25T07:08:10.429 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already in. 2024-04-25T07:08:10.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:08:10.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-04-25T07:08:10.967 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already in. 2024-04-25T07:08:10.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-04-25T07:08:15.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-04-25T07:08:15.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-04-25T07:08:15.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-04-25T07:08:15.987 INFO:tasks.workunit.client.0.smithi088.stdout:writing non-divergent object existing_4 2024-04-25T07:08:15.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:08:16.396 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:08:16.396 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:08:11.712129+0000 29'100 43:126 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:01:04.447439+0000 0 0 2024-04-25T07:08:16.396 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:08:16.396 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:08:16.397 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:08:16.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:08:16.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:08:16.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-04-25T07:08:16.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:08:16.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:08:16.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:08:16.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:08:16.530 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:08:16.530 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:08:16.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:08:16.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:08:16.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:08:16.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:08:16.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:08:16.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:08:16.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:08:16.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:08:16.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:08:17.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:08:17.130 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:08:17.130 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:08:17.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:08:17.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:08:17.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:08:17.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789957 2024-04-25T07:08:17.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789957 2024-04-25T07:08:17.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789957' 2024-04-25T07:08:17.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:08:17.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:08:17.427 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:08:17.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:08:17.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:08:17.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:08:17.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:08:17.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:08:17.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=167503724548 2024-04-25T07:08:17.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 167503724548 2024-04-25T07:08:17.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-04-25T07:08:17.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:08:17.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-158913789957 2024-04-25T07:08:17.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:08:17.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:08:17.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-158913789957 2024-04-25T07:08:17.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:08:17.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789957 2024-04-25T07:08:17.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-04-25T07:08:17.597 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 158913789957 2024-04-25T07:08:17.597 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:08:18.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789955 -lt 158913789957 2024-04-25T07:08:18.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:08:19.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:08:19.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:08:19.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789957 -lt 158913789957 2024-04-25T07:08:19.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:08:19.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-167503724548 2024-04-25T07:08:19.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:08:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:08:19.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-167503724548 2024-04-25T07:08:19.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:08:19.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=167503724548 2024-04-25T07:08:19.487 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 167503724548 2024-04-25T07:08:19.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-04-25T07:08:19.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:08:19.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 167503724548 -lt 167503724548 2024-04-25T07:08:19.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:08:19.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:19.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:08:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:20.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:20.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:20.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:20.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:20.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:20.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:20.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:20.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:20.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:20.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:08:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:08:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:08:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:21.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:08:21.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:21.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:21.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:21.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:21.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:21.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:21.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:21.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:21.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:21.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:21.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:21.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:22.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:23.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:23.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:23.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:08:23.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:23.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:08:23.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:23.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:23.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:23.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:23.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:23.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:23.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:23.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:23.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:23.656 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:23.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:23.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:24.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:24.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:24.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:24.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:24.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:24.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:24.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:24.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:24.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:24.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:24.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:24.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:08:24.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:24.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:08:25.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:25.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:25.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:25.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:25.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:25.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:25.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:25.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:25.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:25.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:25.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:25.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:26.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:26.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:26.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:26.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:26.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:26.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:26.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:08:26.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:26.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:08:27.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:27.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:27.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:27.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:27.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:27.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:27.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:27.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:27.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:27.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:27.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:27.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:28.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:28.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:28.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:28.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:28.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:28.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:28.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:28.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:28.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:28.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:28.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:28.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:08:28.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:28.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:08:30.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:30.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:30.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:30.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:30.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:30.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:30.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:30.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:30.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:30.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:30.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:30.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:31.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:31.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:31.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:31.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:31.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:31.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:31.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:31.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:31.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:32.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:32.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:32.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:08:32.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:32.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:08:35.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:35.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:35.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:35.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:35.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:35.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:35.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:35.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:35.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:35.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:35.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:35.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:36.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:36.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:36.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:36.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:36.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:36.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:36.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:08:36.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:36.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:08:43.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:43.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:43.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:43.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:43.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:43.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:43.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:43.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:43.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:43.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:43.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:43.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:44.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:44.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:44.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:44.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:44.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:44.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:44.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:44.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:44.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:44.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:44.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:44.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:08:44.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:08:44.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:08:52.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:08:52.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:08:52.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:08:52.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:08:52.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:08:52.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:08:52.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:08:52.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:08:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:08:52.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:08:52.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:08:52.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:08:53.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:08:53.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:08:53.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:08:53.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:08:53.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:08:53.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:08:53.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:08:53.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:08:53.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:08:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:08:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:08:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:08:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "a220be49-7975-420a-96b2-3612294c6036", 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-04-25T07:08:54.116689+0000", 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-04-25T07:08:54.114 INFO:tasks.workunit.client.0.smithi088.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 17 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "OSD_DOWN": { 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 osds down", 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.115 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "osd.1 (root=default,host=smithi088) is down" 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:08:54.116 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:08:54.117 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-04-25T07:08:54.118 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:05:16.668935Z", 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T07:05:16.668935Z", 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-04-25T07:08:54.119 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-04-25T07:08:54.120 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.121 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 43, 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T07:08:54.122 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T07:05:17.518865+0000", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:08:11.375865+0000", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-04-25T07:08:08.451526+0000", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-04-25T07:08:10.807218+0000", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5800384, 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "nodown", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "noin", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "noout", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.123 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 10, 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 3, 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T07:08:54.124 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "test", 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T07:07:02.266271+0000", 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 16389, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 3, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 1, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T07:08:54.125 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 1, 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 1, 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 1, 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 1, 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 1, 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T07:08:54.126 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "30", 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "pool", 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 1, 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 30, 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [ 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.127 INFO:tasks.workunit.client.0.smithi088.stdout: "snapid": 1, 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "stamp": "2024-04-25T07:07:45.173379+0000", 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "snap1" 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T07:08:54.128 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "myprofile", 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T07:08:54.129 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 8192, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T07:08:54.130 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {} 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "9e25fde4-eacc-4fda-876f-6ce01e9c6922", 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:08:54.131 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 5, 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 29, 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 37, 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 42, 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 31, 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2206864998 2024-04-25T07:08:54.132 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2206864998 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.133 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2206864998 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2206864998 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:08:54.134 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2206864998 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:08:54.135 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2206864998 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2206864998 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.136 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2206864998 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6803/2206864998", 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6805/2206864998", 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2206864998", 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2206864998", 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.137 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "b02c5640-7b84-4669-94ce-bea6cb51d86c", 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 0, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 10, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 25, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 32, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:08:54.138 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:08:54.139 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:08:54.140 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:08:54.141 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.142 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/2807942619", 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/2807942619", 2024-04-25T07:08:54.143 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2807942619", 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2807942619", 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "exists" 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "5a5fd889-264a-480b-8a6e-79c26bb25857", 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 15, 2024-04-25T07:08:54.144 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 29, 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 39, 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 0, 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 33, 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2144286097 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:08:54.145 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2144286097 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2144286097 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.146 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2144286097 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2144286097 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2144286097 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.147 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2144286097 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2144286097 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.148 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/2144286097", 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/2144286097", 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2144286097", 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2144286097", 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-04-25T07:08:54.149 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:07:56.409218+0000", 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:05:49.327726+0000", 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:07:57.044684+0000", 2024-04-25T07:08:54.150 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:06:24.100582+0000", 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:07:57.590552+0000", 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:08:54.151 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:06:57.548815+0000", 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-04-25T07:08:54.152 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "myprofile": { 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-device-class": "", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-failure-domain": "osd", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-num-failure-domains": "0", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-osds-per-failure-domain": "0", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-root": "default", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "1", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van", 2024-04-25T07:08:54.153 INFO:tasks.workunit.client.0.smithi088.stdout: "w": "8" 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [], 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [], 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-04-25T07:08:54.154 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6804/2206864998,v1:127.0.0.1:6805/2206864998]", 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:08:54.155 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:08:54.156 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:08:54.156 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:08:54.156 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:08:54.156 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:08:54.156 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:08:54.156 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:08:54.157 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:08:54.157 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:08:54.157 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:08:54.157 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:08:54.157 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:08:54.157 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:08:54.157 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:08:54.158 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:08:54.158 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:54.158 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:08:54.158 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:54.158 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:08:54.158 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:05:23.743063Z", 2024-04-25T07:08:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:08:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:08:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:08:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:08:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:08:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:08:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:08:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6802/2206864998,v1:127.0.0.1:6803/2206864998]", 2024-04-25T07:08:54.160 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:08:54.160 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2206864998,v1:127.0.0.1:6809/2206864998]", 2024-04-25T07:08:54.160 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2206864998,v1:127.0.0.1:6807/2206864998]", 2024-04-25T07:08:54.160 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:08:54.160 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:08:54.160 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:08:54.160 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:08:54.161 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:08:54.161 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:08:54.161 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:08:54.161 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:08:54.161 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:08:54.161 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:08:54.161 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:08:54.162 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:08:54.162 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:08:54.162 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.162 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.162 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:08:54.162 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:08:54.163 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/2807942619,v1:127.0.0.1:6813/2807942619]", 2024-04-25T07:08:54.163 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:08:54.163 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:08:54.163 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:08:54.163 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:08:54.163 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:08:54.163 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:08:54.164 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:08:54.164 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:08:54.164 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:08:54.164 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:08:54.164 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:08:54.164 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:08:54.165 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:08:54.165 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:08:54.165 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:08:54.165 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:08:54.165 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:08:54.165 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:08:54.165 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:54.166 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:08:54.166 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:54.166 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:08:54.166 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:05:57.179930Z", 2024-04-25T07:08:54.166 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:08:54.166 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:08:54.167 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:08:54.167 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:08:54.167 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:08:54.167 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:08:54.167 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:08:54.167 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/2807942619,v1:127.0.0.1:6811/2807942619]", 2024-04-25T07:08:54.168 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:08:54.168 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2807942619,v1:127.0.0.1:6817/2807942619]", 2024-04-25T07:08:54.168 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2807942619,v1:127.0.0.1:6815/2807942619]", 2024-04-25T07:08:54.168 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:08:54.168 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:08:54.169 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:08:54.169 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:08:54.169 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:08:54.169 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:08:54.169 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:08:54.169 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:08:54.169 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:08:54.170 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:08:54.170 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:08:54.170 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:08:54.170 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:08:54.170 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.170 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.171 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:08:54.171 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:08:54.171 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/2144286097,v1:127.0.0.1:6813/2144286097]", 2024-04-25T07:08:54.171 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:08:54.171 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:08:54.171 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:08:54.172 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:08:54.172 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:08:54.172 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:08:54.172 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:08:54.172 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:08:54.172 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:08:54.173 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:08:54.173 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:08:54.173 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:08:54.173 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:08:54.173 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:08:54.173 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:08:54.173 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:08:54.174 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:08:54.174 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:08:54.174 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:54.174 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:08:54.174 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:08:54.174 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:08:54.175 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:06:31.101956Z", 2024-04-25T07:08:54.175 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:08:54.175 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:08:54.175 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:08:54.175 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:08:54.175 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:08:54.176 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:08:54.176 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:08:54.176 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/2144286097,v1:127.0.0.1:6811/2144286097]", 2024-04-25T07:08:54.176 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:08:54.176 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2144286097,v1:127.0.0.1:6817/2144286097]", 2024-04-25T07:08:54.176 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2144286097,v1:127.0.0.1:6815/2144286097]", 2024-04-25T07:08:54.177 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:08:54.177 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:08:54.177 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:08:54.177 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:08:54.177 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:08:54.177 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:08:54.177 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:08:54.177 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:08:54.178 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:08:54.178 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:08:54.178 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:08:54.178 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:08:54.178 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:08:54.178 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.179 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.179 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-04-25T07:08:54.179 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:08:54.179 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-04-25T07:08:54.179 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [] 2024-04-25T07:08:54.179 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.179 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-04-25T07:08:54.180 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.180 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:08:54.180 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 37 2024-04-25T07:08:54.180 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.180 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.180 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:08:54.180 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 39 2024-04-25T07:08:54.181 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.181 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.181 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.181 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-04-25T07:08:54.181 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 43, 2024-04-25T07:08:54.181 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-04-25T07:08:54.181 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-04-25T07:08:54.182 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.182 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:08:54.182 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-04-25T07:08:54.182 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:08:54.182 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.183 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.183 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:08:54.183 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.1", 2024-04-25T07:08:54.183 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:08:54.183 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.183 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.183 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:08:54.184 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.2", 2024-04-25T07:08:54.184 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:08:54.184 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.184 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.184 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-04-25T07:08:54.184 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.185 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-04-25T07:08:54.185 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-04-25T07:08:54.185 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.185 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.185 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:08:54.185 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-04-25T07:08:54.185 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.185 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.186 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-04-25T07:08:54.186 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-04-25T07:08:54.186 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.186 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.186 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-04-25T07:08:54.186 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-04-25T07:08:54.186 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.186 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.187 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-04-25T07:08:54.187 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-04-25T07:08:54.187 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.187 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.187 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-04-25T07:08:54.187 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-04-25T07:08:54.187 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.187 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.188 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-04-25T07:08:54.188 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-04-25T07:08:54.188 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.188 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.188 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-04-25T07:08:54.188 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-04-25T07:08:54.188 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.188 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.189 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-04-25T07:08:54.189 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-04-25T07:08:54.189 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.189 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.189 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-04-25T07:08:54.189 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-04-25T07:08:54.189 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.190 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.190 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-04-25T07:08:54.190 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-04-25T07:08:54.190 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.190 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.190 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:08:54.190 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-04-25T07:08:54.191 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.191 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.191 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-04-25T07:08:54.191 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.191 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-04-25T07:08:54.191 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-04-25T07:08:54.191 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:08:54.191 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T07:08:54.192 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:08:54.192 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:08:54.192 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:08:54.192 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:08:54.192 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.192 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T07:08:54.192 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:08:54.193 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:08:54.193 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.193 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.193 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.193 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.193 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-04-25T07:08:54.193 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-04-25T07:08:54.193 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:08:54.194 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T07:08:54.194 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:08:54.194 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:08:54.194 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:08:54.194 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:08:54.194 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.194 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T07:08:54.195 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:08:54.195 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:08:54.195 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.195 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.195 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.195 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.195 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T07:08:54.196 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-04-25T07:08:54.196 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:08:54.196 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T07:08:54.196 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:08:54.196 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:08:54.196 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:08:54.196 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:08:54.197 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.197 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:08:54.197 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:08:54.197 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:08:54.197 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.197 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.197 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:08:54.197 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:08:54.198 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T07:08:54.198 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.198 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.198 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:08:54.198 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:08:54.198 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T07:08:54.198 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:08:54.199 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.200 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.201 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 1, 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "test", 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 3, 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.202 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "set_chooseleaf_tries", 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 5 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "set_choose_tries", 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 100 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_indep", 2024-04-25T07:08:54.203 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:08:54.204 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 1, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-04-25T07:08:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-04-25T07:08:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-04-25T07:08:54.207 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 5, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 3 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 1, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 3, 2024-04-25T07:08:54.208 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 62667, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 101, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 1, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 303, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 101, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 101, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T07:08:54.209 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 101, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 101, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T07:08:54.210 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T07:08:54.211 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 18, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 18, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 3, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 3, 2024-04-25T07:08:54.212 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 83672, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2164, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81461, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314489128, 2024-04-25T07:08:54.213 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036867072, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2215936, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 1550569, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83416932 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T07:08:54.214 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-04-25T07:08:54.215 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 83672, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2164, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 10, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 81461, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314489128, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322036867072, 2024-04-25T07:08:54.216 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2215936, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 1550569, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 10396, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 83416932 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T07:08:54.217 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.218 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 62667, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 101, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 1, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 303, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 101, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 101, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T07:08:54.219 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 101, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 101, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T07:08:54.220 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T07:08:54.221 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1241088, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 1241088, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 18, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 18, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 3, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 3, 2024-04-25T07:08:54.222 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 3 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 158913789964 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 42949672992 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 167503724555 2024-04-25T07:08:54.223 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 1 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2147483647, 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.224 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 1, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T07:08:54.225 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 0, 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 5, 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:07:58.675780+0000", 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-04-25T07:08:54.226 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 97, 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 399, 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 547, 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:08:54.227 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:08:54.228 INFO:tasks.workunit.client.0.smithi088.stderr:report 2246766656 2024-04-25T07:08:54.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:08:54.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:08:54.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-04-25T07:08:54.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:08:54.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:08:54.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:08:54.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:08:54.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:08:54.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:08:54.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-04-25T07:08:54.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:08:54.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:08:54.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-25T07:08:54.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:08:54.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-04-25T07:08:54.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-04-25T07:08:55.382 INFO:tasks.workunit.client.0.smithi088.stderr:PG '1.0' not found 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:55.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:55.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:08:55.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:08:55.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:55.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:08:55.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:55.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:08:55.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:55.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:55.921 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:55.921 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:55.921 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:08:55.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:08:55.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:08:55.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:55.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:55.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:55.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:55.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:55.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:55.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:55.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:08:55.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:08:55.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:08:55.923 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:08:55.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:08:55.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:55.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:08:55.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:08:55.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:08:55.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:08:55.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:08:55.973 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:55.977+0000 7f1baf2a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:55.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:55.977+0000 7f1baf2a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:55.978 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:55.981+0000 7f1baf2a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:56.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:08:56.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:56.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:08:56.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:56.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:56.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:56.367 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:08:56.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:56.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:56.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:56.530 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:56.533+0000 7f1baf2a47c0 -1 Falling back to public interface 2024-04-25T07:08:56.841 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 37 up_thru 42 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/2206864998,v1:127.0.0.1:6803/2206864998] [v2:127.0.0.1:6804/2206864998,v1:127.0.0.1:6805/2206864998] exists,up 9e25fde4-eacc-4fda-876f-6ce01e9c6922 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:08:56.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:08:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:08:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-04-25T07:08:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:08:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:08:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-25T07:08:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:08:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-04-25T07:08:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-04-25T07:08:58.032 INFO:tasks.workunit.client.0.smithi088.stderr:PG '1.0' not found 2024-04-25T07:08:58.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:08:58.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:08:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:58.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:58.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:58.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:58.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:58.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:58.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:08:58.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:08:58.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:08:58.588 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:08:58.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:08:58.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:58.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:08:58.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:08:58.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:08:58.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:08:58.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:08:58.628 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:58.629+0000 7f1f8a8ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:58.630 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:58.633+0000 7f1f8a8ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:58.631 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:58.633+0000 7f1f8a8ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:59.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:08:59.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:59.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:08:59.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:59.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:59.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:59.012 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:08:59.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:59.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:59.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:59.076 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:59.077+0000 7f1baf2a47c0 -1 osd.0 43 log_to_monitors true 2024-04-25T07:08:59.194 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:08:59.197+0000 7f1f8a8ab7c0 -1 Falling back to public interface 2024-04-25T07:08:59.449 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/2144286097,v1:127.0.0.1:6811/2144286097] [v2:127.0.0.1:6812/2144286097,v1:127.0.0.1:6813/2144286097] exists,up 5a5fd889-264a-480b-8a6e-79c26bb25857 2024-04-25T07:08:59.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:08:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-04-25T07:09:01.049 INFO:tasks.workunit.client.0.smithi088.stderr:PG '1.0' not found 2024-04-25T07:09:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:09:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:09:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-04-25T07:09:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:09:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:09:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:09:01.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:09:01.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:09:01.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:09:01.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:09:01.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:09:01.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:09:01.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:09:01.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:09:01.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:09:01.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:09:01.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:09:01.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:09:01.695 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:01.697+0000 7f1f8a8ab7c0 -1 osd.2 43 log_to_monitors true 2024-04-25T07:09:02.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:09:02.207 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:09:02.207 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:09:02.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:09:02.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:02.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:09:02.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495619 2024-04-25T07:09:02.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495619 2024-04-25T07:09:02.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-197568495619' 2024-04-25T07:09:02.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:02.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:09:02.544 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:09:02.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:09:02.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:09:02.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:09:02.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:02.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:09:02.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364803 2024-04-25T07:09:02.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364803 2024-04-25T07:09:02.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-197568495619 2-214748364803' 2024-04-25T07:09:02.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:02.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-197568495619 2024-04-25T07:09:02.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:02.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:09:02.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-197568495619 2024-04-25T07:09:02.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:02.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495619 2024-04-25T07:09:02.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-04-25T07:09:02.714 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 197568495619 2024-04-25T07:09:02.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:09:03.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495619 -lt 197568495619 2024-04-25T07:09:03.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:03.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-214748364803 2024-04-25T07:09:03.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:03.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:09:03.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-214748364803 2024-04-25T07:09:03.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:03.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364803 2024-04-25T07:09:03.187 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 214748364803 2024-04-25T07:09:03.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-04-25T07:09:03.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:09:03.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364802 -lt 214748364803 2024-04-25T07:09:03.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:09:04.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:09:04.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:09:05.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364803 -lt 214748364803 2024-04-25T07:09:05.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:09:05.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:05.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:05.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:09:05.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:05.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:05.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:05.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:05.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:05.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:05.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:06.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:06.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:06.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:06.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:06.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:06.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:09:06.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:09:06.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:09:06.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:09:06.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:09:06.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:09:06.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:06.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:06.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:06.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:06.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:06.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:06.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:07.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:07.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:07.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:07.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:07.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:07.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:09:07.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:09:07.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:09:07.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:09:07.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:09:07.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:09:07.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:09:07.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:09:08.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:09:08.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:09:08.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:09:08.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:09:08.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:09:08.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:09:08.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:08.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:08.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:08.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:08.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:08.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:08.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:08.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:08.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:08.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:08.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:09:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:09:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:09:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:09:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:09:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:09:09.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:09:09.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:09:10.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:09:10.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:09:10.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:09:10.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:09:10.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:09:10.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:09:10.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:10.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:10.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:10.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:10.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:10.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:10.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:10.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:10.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:10.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:10.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:11.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:11.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:09:11.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:09:11.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:09:11.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:09:11.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:09:11.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:09:11.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:09:11.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:09:11.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:09:11.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:09:11.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:09:11.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:09:11.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:09:12.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:09:12.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:12.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:12.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:12.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:12.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:12.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:12.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:13.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:13.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:13.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:13.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:13.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:13.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:09:13.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:09:13.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:09:13.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:09:13.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:09:13.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:09:13.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:09:13.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:09:14.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:09:14.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:09:14.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:09:14.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:09:14.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:09:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:09:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:15.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:15.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:15.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:15.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:15.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:15.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:16.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:16.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:16.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:16.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:16.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:16.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:09:16.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:09:16.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:09:16.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:09:16.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:09:16.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:09:16.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:09:16.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:09:17.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:09:17.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:09:17.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:09:17.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:09:17.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:09:20.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:09:20.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:20.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:20.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:20.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:20.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:20.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:20.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:21.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:21.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:21.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:21.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:21.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:21.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:09:21.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:09:21.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:09:21.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:09:21.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:09:21.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:09:21.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:09:21.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:09:22.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:09:22.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:09:22.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:09:22.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:09:22.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:09:28.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:09:28.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:28.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:28.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:28.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:28.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:28.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:28.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:28.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:28.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:28.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:28.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:29.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:29.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:09:29.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:09:29.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:09:29.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:09:29.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:09:29.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:09:29.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:09:29.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:09:30.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:09:30.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:09:30.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:09:30.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:09:30.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:09:37.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:09:37.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:37.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:37.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:37.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:37.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:37.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:37.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:37.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:09:37.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:37.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:37.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:38.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:09:38.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:09:38.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:09:38.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:09:38.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:09:38.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:09:38.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:09:38.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:09:38.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:09:38.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:09:38.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:09:38.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:09:38.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:09:39.477 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "a220be49-7975-420a-96b2-3612294c6036", 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-04-25T07:09:39.480424+0000", 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_WARN", 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:09:39.478 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 17 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "OSD_DOWN": { 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 osds down", 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:09:39.479 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "osd.1 (root=default,host=smithi088) is down" 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.480 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:09:39.481 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-04-25T07:09:39.482 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:05:16.668935Z", 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T07:05:16.668935Z", 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-04-25T07:09:39.483 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.484 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-04-25T07:09:39.485 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 51, 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "c26ef714-38b1-44b0-8913-6c7fa77cb52c", 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T07:05:17.518865+0000", 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:09:02.409695+0000", 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-04-25T07:09:02.099947+0000", 2024-04-25T07:09:39.486 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-04-25T07:08:10.807218+0000", 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5800384, 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "nodown", 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "noin", 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "noout", 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 12, 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:09:39.487 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 1, 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 3, 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "jewel", 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "test", 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T07:07:02.266271+0000", 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 16389, 2024-04-25T07:09:39.488 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 3, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 1, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 1, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 1, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 1, 2024-04-25T07:09:39.489 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 1, 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 1, 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "30", 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:09:39.490 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "pool", 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 1, 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 30, 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [ 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "snapid": 1, 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "stamp": "2024-04-25T07:07:45.173379+0000", 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "snap1" 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T07:09:39.491 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "myprofile", 2024-04-25T07:09:39.492 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 8192, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T07:09:39.493 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {} 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "9e25fde4-eacc-4fda-876f-6ce01e9c6922", 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 37, 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 43, 2024-04-25T07:09:39.494 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 46, 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 50, 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 45, 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1243882787 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1243882787 2024-04-25T07:09:39.495 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1243882787 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1243882787 2024-04-25T07:09:39.496 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1243882787 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1243882787 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.497 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1243882787 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1243882787 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.498 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6803/1243882787", 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6805/1243882787", 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1243882787", 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1243882787", 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "b02c5640-7b84-4669-94ce-bea6cb51d86c", 2024-04-25T07:09:39.499 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 0, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 10, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 25, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 32, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:09:39.500 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.501 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.502 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.503 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2807942619 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/2807942619", 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/2807942619", 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2807942619", 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2807942619", 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: "exists" 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.504 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "5a5fd889-264a-480b-8a6e-79c26bb25857", 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 39, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 43, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 50, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 0, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 49, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.505 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 466927137 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 466927137 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.506 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 466927137 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 466927137 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.507 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 466927137 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 466927137 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.508 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 466927137 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 466927137 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/466927137", 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/466927137", 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/466927137", 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/466927137", 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T07:09:39.509 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:08:59.429786+0000", 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:09:39.510 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:05:49.327726+0000", 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:07:57.044684+0000", 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:06:24.100582+0000", 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:09:39.511 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "2024-04-25T07:09:01.952851+0000", 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T07:06:57.548815+0000", 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [], 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-04-25T07:09:39.512 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "myprofile": { 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-device-class": "", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-failure-domain": "osd", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-num-failure-domains": "0", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-osds-per-failure-domain": "0", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "crush-root": "default", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "1", 2024-04-25T07:09:39.513 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van", 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "w": "8" 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [], 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [], 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:09:39.514 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6804/1243882787,v1:127.0.0.1:6805/1243882787]", 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:09:39.515 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:09:39.516 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:05:23.743063Z", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6802/1243882787,v1:127.0.0.1:6803/1243882787]", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1243882787,v1:127.0.0.1:6809/1243882787]", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1243882787,v1:127.0.0.1:6807/1243882787]", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:09:39.517 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/2807942619,v1:127.0.0.1:6813/2807942619]", 2024-04-25T07:09:39.518 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:09:39.519 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:05:57.179930Z", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/2807942619,v1:127.0.0.1:6811/2807942619]", 2024-04-25T07:09:39.520 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2807942619,v1:127.0.0.1:6817/2807942619]", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2807942619,v1:127.0.0.1:6815/2807942619]", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:09:39.521 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/466927137,v1:127.0.0.1:6813/466927137]", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:09:39.522 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T07:06:31.101956Z", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T07:09:39.523 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/466927137,v1:127.0.0.1:6811/466927137]", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/466927137,v1:127.0.0.1:6817/466927137]", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/466927137,v1:127.0.0.1:6815/466927137]", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:09:39.524 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 26, 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [ 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.525 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: "floor": 26 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 46 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 51 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.526 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 51, 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.1", 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.2", 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.528 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-04-25T07:09:39.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.530 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-04-25T07:09:39.531 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.532 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.533 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:09:39.534 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-04-25T07:09:39.535 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 19206, 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:09:39.536 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T07:09:39.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 1, 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "test", 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 3, 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "set_chooseleaf_tries", 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 5 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "set_choose_tries", 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 100 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.539 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_indep", 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-04-25T07:09:39.540 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-04-25T07:09:39.541 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 1, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-04-25T07:09:39.542 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-04-25T07:09:39.543 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 5, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 3 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 1, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 3, 2024-04-25T07:09:39.544 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 62667, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 101, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 1, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 303, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 101, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 101, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 101, 2024-04-25T07:09:39.545 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 101, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T07:09:39.546 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.547 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 18, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 18, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 3, 2024-04-25T07:09:39.548 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 3, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 85100, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2292, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 15, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 82736, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314487700, 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T07:09:39.549 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322035404800, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2347008, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 1605477, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 16263, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 84721785 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T07:09:39.550 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T07:09:39.551 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 3, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 3, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 3, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 314572800, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 85100, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 2292, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 15, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 82736, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 314487700, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 322122547200, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 322035404800, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:09:39.552 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 2347008, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 1605477, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 16263, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 84721785 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 0, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [], 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 1 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.553 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T07:09:39.554 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 62667, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 101, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 1, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 303, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 101, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 101, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 101, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 101, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T07:09:39.555 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T07:09:39.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1241088, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 1241088, 2024-04-25T07:09:39.557 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 18, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 18, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 3, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 3, 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 3 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.558 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 197568495626 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 42949672992 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 214748364810 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+undersized+degraded", 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 1 2024-04-25T07:09:39.559 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2147483647, 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 1, 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 1, 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 0, 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 0, 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:09:39.561 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 5, 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T07:07:58.675780+0000", 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 120, 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 483, 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T07:09:39.562 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 662, 2024-04-25T07:09:39.563 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-04-25T07:09:39.563 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-04-25T07:09:39.563 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:09:39.563 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:09:39.563 INFO:tasks.workunit.client.0.smithi088.stderr:report 646526065 2024-04-25T07:09:39.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:09:39.564 INFO:tasks.workunit.client.0.smithi088.stdout:delay recovery 2024-04-25T07:09:39.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-04-25T07:09:39.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:09:39.888 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:39.888 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:09:02.687449+0000 43'102 51:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:09:39.889 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:09:39.889 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:39.889 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:09:39.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:09:39.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-04-25T07:09:39.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:09:39.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:09:39.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:39.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:39.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:09:39.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.0.asok 2024-04-25T07:09:39.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:09:39.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:09:40.004 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:09:40.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:09:40.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-04-25T07:09:40.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:09:40.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:09:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:40.017 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:09:40.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.2.asok 2024-04-25T07:09:40.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:09:40.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:09:40.115 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:09:40.127 INFO:tasks.workunit.client.0.smithi088.stdout:revive divergent 1 2024-04-25T07:09:40.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-04-25T07:09:40.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:09:40.522 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:40.522 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:09:02.687449+0000 43'102 51:139 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:09:40.522 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:09:40.522 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:40.522 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:09:40.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-04-25T07:09:41.291 INFO:tasks.workunit.client.0.smithi088.stderr:noup is set 2024-04-25T07:09:41.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-04-25T07:09:41.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:09:41.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:09:41.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:09:41.355 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:09:41.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:09:41.357 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:41.356+0000 7fb466f017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:41.360+0000 7fb466f017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:41.359 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:41.360+0000 7fb466f017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:41.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:09:41.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-04-25T07:09:41.899 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:41.900+0000 7fb466f017c0 -1 Falling back to public interface 2024-04-25T07:09:44.399 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:44.400+0000 7fb466f017c0 -1 osd.1 30 log_to_monitors true 2024-04-25T07:09:46.744 INFO:tasks.workunit.client.0.smithi088.stdout:delay recovery divergent 2024-04-25T07:09:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-04-25T07:09:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:09:47.164 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:47.164 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:09:02.687449+0000 43'102 53:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:30:27.976708+0000 0 0 2024-04-25T07:09:47.164 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:09:47.164 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:47.164 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:09:47.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-04-25T07:09:47.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:09:47.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:09:47.181 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:47.181 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:47.182 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:09:47.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26017/ceph-osd.1.asok 2024-04-25T07:09:47.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:09:47.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26017/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:09:47.301 INFO:tasks.workunit.client.0.smithi088.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:09:47.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-04-25T07:09:47.943 INFO:tasks.workunit.client.0.smithi088.stderr:noup is unset 2024-04-25T07:09:47.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-04-25T07:09:47.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:09:47.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:09:47.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:09:47.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:09:47.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:47.963 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:09:47.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:09:47.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:47.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:09:48.541 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 46 up_thru 50 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/1243882787,v1:127.0.0.1:6803/1243882787] [v2:127.0.0.1:6804/1243882787,v1:127.0.0.1:6805/1243882787] exists,up 9e25fde4-eacc-4fda-876f-6ce01e9c6922 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:09:48.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:48.544 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:09:48.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:09:48.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:48.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:48.976 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/124774100,v1:127.0.0.1:6819/124774100] [v2:127.0.0.1:6820/124774100,v1:127.0.0.1:6821/124774100] exists,up b02c5640-7b84-4669-94ce-bea6cb51d86c 2024-04-25T07:09:48.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:48.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:48.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:48.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-04-25T07:09:48.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:09:48.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:09:48.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:09:48.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:09:48.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:48.978 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:09:48.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:09:48.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:48.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:49.412 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/466927137,v1:127.0.0.1:6811/466927137] [v2:127.0.0.1:6812/466927137,v1:127.0.0.1:6813/466927137] exists,up 5a5fd889-264a-480b-8a6e-79c26bb25857 2024-04-25T07:09:49.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:49.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:49.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:49.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:09:49.806 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:49.806 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:09:02.687449+0000 43'102 53:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:30:27.976708+0000 0 0 2024-04-25T07:09:49.806 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:09:49.806 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:49.806 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:09:49.821 INFO:tasks.workunit.client.0.smithi088.stdout:wait for peering 2024-04-25T07:09:49.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-04-25T07:09:49.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:09:50.216 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:50.216 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:09:02.687449+0000 43'102 53:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:30:27.976708+0000 0 0 2024-04-25T07:09:50.217 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:09:50.217 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:50.217 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:09:50.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-04-25T07:09:50.363 INFO:tasks.workunit.client.0.smithi088.stdout:killing divergent 1 2024-04-25T07:09:50.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-04-25T07:09:50.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:09:50.754 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:50.754 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:09:49.224113+0000 43'102 57:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:09:50.754 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:09:50.754 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:50.754 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:09:50.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:09:50.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:09:50.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:09:50.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:09:50.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:09:50.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:09:50.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:09:50.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-04-25T07:09:50.881 INFO:tasks.workunit.client.0.smithi088.stdout:reviving divergent 1 2024-04-25T07:09:50.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:09:51.280 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:51.281 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:09:49.224113+0000 43'102 57:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:09:51.281 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:09:51.281 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:51.281 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:09:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-04-25T07:09:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:09:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:09:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:09:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:09:51.298 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:09:51.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:51.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:09:51.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:09:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' 2024-04-25T07:09:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:09:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:09:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:09:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:09:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:09:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:09:51.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:09:51.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:09:51.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:09:51.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:09:51.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:09:51.301 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:09:51.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=c26ef714-38b1-44b0-8913-6c7fa77cb52c --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:09:51.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:09:51.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:09:51.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:09:51.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:09:51.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:09:51.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:51.348+0000 7fb2d59b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:51.381 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:51.384+0000 7fb2d59b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:51.382 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:51.384+0000 7fb2d59b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:51.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:09:51.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:09:51.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:09:51.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:09:51.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:09:51.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:09:51.708 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:09:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:51.943 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:51.944+0000 7fb2d59b57c0 -1 Falling back to public interface 2024-04-25T07:09:52.115 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/124774100,v1:127.0.0.1:6819/124774100] [v2:127.0.0.1:6820/124774100,v1:127.0.0.1:6821/124774100] exists,up b02c5640-7b84-4669-94ce-bea6cb51d86c 2024-04-25T07:09:52.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:52.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:52.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:52.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-04-25T07:09:54.738 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:09:54.740+0000 7fb2d59b57c0 -1 osd.1 57 log_to_monitors true 2024-04-25T07:10:12.118 INFO:tasks.workunit.client.0.smithi088.stdout:allowing recovery 2024-04-25T07:10:12.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-04-25T07:10:12.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:10:12.539 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:12.539 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-04-25T07:09:56.517703+0000 57'103 62:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:29:33.509213+0000 0 0 2024-04-25T07:10:12.539 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:10:12.539 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:12.540 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:10:12.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:10:12.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:10:12.710 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:10:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:10:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:10:12.874 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:10:12.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:10:12.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:10:13.042 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:10:13.056 INFO:tasks.workunit.client.0.smithi088.stdout:reading divergent objects 2024-04-25T07:10:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-04-25T07:10:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:10:13.460 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:13.460 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-04-25T07:09:56.517703+0000 57'103 62:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:07:02.332052+0000 0'0 2024-04-25T07:07:02.332052+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:29:33.509213+0000 0 0 2024-04-25T07:10:13.460 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:10:13.460 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:13.460 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:10:13.477 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-04-25T07:10:13.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-04-25T07:10:13.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:10:13.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:10:13.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:10:13.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:10:13.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:10:13.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:10:13.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:10:13.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:10:13.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-04-25T07:10:13.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-04-25T07:10:13.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:10:13.632 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-04-25T07:07:37.406254+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:10:13.632 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-04-25T07:07:37.406254+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:10:13.632 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:10:13.632 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-04-25T07:10:13.632 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-04-25T07:10:13.632 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4647.0:1 2024-04-25T07:07:45.463989+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4647.0:1 2024-04-25T07:07:45.463989+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4647.0:1 2024-04-25T07:07:45.463989+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:10:13.633 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4638.0:1 2024-04-25T07:07:44.168569+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4644.0:1 2024-04-25T07:07:45.175522+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4638.0:1 2024-04-25T07:07:44.168569+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4644.0:1 2024-04-25T07:07:45.175522+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4644.0:1 2024-04-25T07:07:45.175522+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:10:13.635 INFO:tasks.workunit.client.0.smithi088.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:09:49.152+0000 7fb44ae8b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-04-25T07:10:13.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-04-25T07:10:13.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:10:13.641 INFO:tasks.workunit.client.0.smithi088.stdout:success 2024-04-25T07:10:13.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-04-25T07:10:13.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-04-25T07:10:13.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:10:13.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:10:14.133 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T07:10:14.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-04-25T07:10:14.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:10:14.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:10:14.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:10:14.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:10:14.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:10:19.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:10:19.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:10:19.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:10:19.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:10:19.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:10:19.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:10:19.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:10:19.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:10:19.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:10:19.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:10:19.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:10:19.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:10:19.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:10:19.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:10:19.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:10:19.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:10:19.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:10:19.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:10:19.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:10:19.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:10:19.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:10:19.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:10:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:10:19.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:10:19.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:10:19.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:19.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:10:19.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T07:10:19.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:10:19.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:10:19.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:10:19.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/divergent-priors 0 2024-04-25T07:10:19.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:10:19.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:10:19.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:10:19.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:10:19.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:10:19.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:10:19.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:10:19.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:10:19.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:10:19.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:10:19.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:10:19.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:10:19.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:10:19.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:10:19.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:10:19.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:10:19.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:10:19.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:10:19.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:10:19.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:10:19.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:10:19.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:10:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:10:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26017 2024-04-25T07:10:19.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26017 2024-04-25T07:10:19.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:10:19.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:10:19.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:10:19.526 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:10:19.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:10:19.583 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-04-25T07:10:19.583 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-04-25T07:10:19.634 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:10:19.639 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:10:19.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:10:19.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:10:19.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:10:19.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/ec-error-rollforward 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-04-25T07:10:19.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-04-25T07:10:19.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none ' 2024-04-25T07:10:19.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-04-25T07:10:19.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-04-25T07:10:19.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-04-25T07:10:19.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-04-25T07:10:19.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-04-25T07:10:19.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-04-25T07:10:19.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-04-25T07:10:19.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-04-25T07:10:19.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:10:19.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:10:19.646 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:10:19.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:10:19.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:10:19.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:10:19.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:10:19.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:10:19.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:10:19.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:10:19.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:10:19.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:10:19.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:10:19.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:10:19.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:10:19.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:10:19.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:10:19.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:10:19.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:10:19.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:10:19.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:10:19.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:19.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:10:19.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.82769 2024-04-25T07:10:19.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:10:19.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:10:19.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-04-25T07:10:19.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:10:19.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:19.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:10:19.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.82769 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:10:19.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:10:19.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-04-25T07:10:19.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-04-25T07:10:20.156 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:10:20.157 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:20.157 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:20.158 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:20.158 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:20.158 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:10:20.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:10:20.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:10:20.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:10:20.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:10:20.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:10:20.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:10:20.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:10:20.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:10:20.362 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:10:20.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:10:20.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:10:20.363 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:10:20.363 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:20.363 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:10:20.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.82769/ceph-mon.a.asok 2024-04-25T07:10:20.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:10:20.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.82769/ceph-mon.a.asok config get fsid 2024-04-25T07:10:20.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:10:20.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:10:20.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:10:20.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:10:20.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:10:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:10:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:10:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:10:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:10:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:10:20.482 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.82769/ceph-mon.a.asok 2024-04-25T07:10:20.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:10:20.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.82769/ceph-mon.a.asok config get mon_host 2024-04-25T07:10:20.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-04-25T07:10:20.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-04-25T07:10:20.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:10:20.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:10:20.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:10:20.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-04-25T07:10:20.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:10:21.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:10:21.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:21.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:10:21.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:10:21.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:10:21.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:10:21.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-04-25T07:10:21.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:10:21.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:10:21.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:10:21.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:10:21.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-04-25T07:10:21.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:21.518 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:10:21.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:10:21.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:10:21.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:21.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:10:21.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:10:21.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:10:21.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:10:21.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:21.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:21.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:21.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:21.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:10:21.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:10:21.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-04-25T07:10:21.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:10:21.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57d13c1b-e6f1-4fcf-9598-57d0d0cf436b 2024-04-25T07:10:21.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 57d13c1b-e6f1-4fcf-9598-57d0d0cf436b' 2024-04-25T07:10:21.521 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 57d13c1b-e6f1-4fcf-9598-57d0d0cf436b 2024-04-25T07:10:21.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:10:21.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDdASpmwysaIBAA26esmWUe/zqTX/OO6UdH0Q== 2024-04-25T07:10:21.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDdASpmwysaIBAA26esmWUe/zqTX/OO6UdH0Q=="}' 2024-04-25T07:10:21.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57d13c1b-e6f1-4fcf-9598-57d0d0cf436b -i td/ec-error-rollforward/0/new.json 2024-04-25T07:10:21.804 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:10:21.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-04-25T07:10:21.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdASpmwysaIBAA26esmWUe/zqTX/OO6UdH0Q== --osd-uuid 57d13c1b-e6f1-4fcf-9598-57d0d0cf436b 2024-04-25T07:10:21.845 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:21.847+0000 7f5223c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:21.873 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:21.875+0000 7f5223c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:21.874 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:21.875+0000 7f5223c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:21.875 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:21.875+0000 7f5223c4b7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:10:21.875 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:21.875+0000 7f5223c4b7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:10:21.875 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:21.875+0000 7f5223c4b7c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:10:21.913 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:21.915+0000 7f5223c4b7c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-04-25T07:10:27.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-04-25T07:10:27.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:10:27.874 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:10:27.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:10:27.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:10:28.447 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:10:28.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:10:28.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:10:28.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:10:28.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:10:28.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:10:28.477 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:28.479+0000 7f761e7657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:28.479 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:28.479+0000 7f761e7657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:28.480 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:28.483+0000 7f761e7657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:28.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:10:28.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:10:28.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:10:28.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:10:28.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:10:28.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:28.901 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:10:28.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:10:28.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:28.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:29.061 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:29.063+0000 7f761e7657c0 -1 Falling back to public interface 2024-04-25T07:10:29.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:30.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:30.320 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:10:30.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:30.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:10:30.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:30.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:30.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:31.611 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:31.611+0000 7f761e7657c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:10:31.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:31.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:31.795 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:10:31.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:10:31.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:31.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:32.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:33.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:33.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:33.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:10:33.330 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:10:33.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:33.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:33.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:34.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:34.777 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:10:34.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:34.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:10:34.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:34.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:35.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:36.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:36.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:36.180 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:10:36.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:10:36.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:36.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:36.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:37.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:37.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:37.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:10:37.708 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:10:37.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:37.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:38.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:39.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:39.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:39.145 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:10:39.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:10:39.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:39.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:39.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:40.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:40.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:40.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:10:40.609 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:10:40.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:40.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:41.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:42.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:42.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:42.059 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:10:42.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:10:42.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:42.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:42.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:43.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:43.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:43.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:10:43.486 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:10:43.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:43.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:44.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:45.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:45.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:45.288 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:10:45.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:10:45.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:45.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:45.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:46.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:46.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:46.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:10:46.721 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:10:46.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:46.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:47.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:48.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:48.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:48.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:10:48.419 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:10:48.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:48.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:49.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:50.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:50.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:50.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:10:50.351 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:10:50.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:50.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:51.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:52.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:52.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:52.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:10:52.358 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:10:52.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:52.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:52.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:53.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:53.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:53.919 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:10:53.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:10:53.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:53.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:54.353 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1038428475,v1:127.0.0.1:6803/1038428475] [v2:127.0.0.1:6804/1038428475,v1:127.0.0.1:6805/1038428475] exists,up 57d13c1b-e6f1-4fcf-9598-57d0d0cf436b 2024-04-25T07:10:54.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:10:54.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:10:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-04-25T07:10:54.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:10:54.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6bd8c733-af70-4894-adc8-d566b8f480e7 2024-04-25T07:10:54.360 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 6bd8c733-af70-4894-adc8-d566b8f480e7 2024-04-25T07:10:54.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6bd8c733-af70-4894-adc8-d566b8f480e7' 2024-04-25T07:10:54.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:10:54.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+ASpm4KieFhAAY5E+SMDOVxbBWkb6M8yLDQ== 2024-04-25T07:10:54.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+ASpm4KieFhAAY5E+SMDOVxbBWkb6M8yLDQ=="}' 2024-04-25T07:10:54.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6bd8c733-af70-4894-adc8-d566b8f480e7 -i td/ec-error-rollforward/1/new.json 2024-04-25T07:10:54.913 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:10:54.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-04-25T07:10:54.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+ASpm4KieFhAAY5E+SMDOVxbBWkb6M8yLDQ== --osd-uuid 6bd8c733-af70-4894-adc8-d566b8f480e7 2024-04-25T07:10:54.962 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:54.963+0000 7f43192677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:54.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:54.971+0000 7f43192677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:54.972 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:54.975+0000 7f43192677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:54.973 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:54.975+0000 7f43192677c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:10:54.973 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:54.975+0000 7f43192677c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:10:54.973 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:54.975+0000 7f43192677c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:10:55.011 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:10:55.011+0000 7f43192677c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-04-25T07:11:00.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-04-25T07:11:00.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:11:00.531 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:11:00.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:11:00.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:11:01.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:11:01.082 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:11:01.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:11:01.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:11:01.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:11:01.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:11:01.111 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:01.111+0000 7f2340da97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:01.143 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:01.143+0000 7f2340da97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:01.144 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:01.147+0000 7f2340da97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:01.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:11:01.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:01.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:11:01.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:01.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:01.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:01.489 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:11:01.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:01.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:01.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:01.710 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:01.711+0000 7f2340da97c0 -1 Falling back to public interface 2024-04-25T07:11:01.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:02.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:02.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:02.909 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:11:02.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:11:02.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:02.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:03.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:04.238 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:04.239+0000 7f2340da97c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:11:04.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:04.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:04.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:11:04.317 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:11:04.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:04.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:04.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:05.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:05.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:05.971 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:11:05.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:11:05.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:05.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:06.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:07.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:07.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:07.404 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:11:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:11:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:07.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:08.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:08.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:08.855 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:11:08.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:11:08.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:08.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:09.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:10.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:10.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:10.420 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:11:10.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:11:10.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:10.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:10.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:11.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:11.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:11.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:11:11.925 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:11:11.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:11.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:12.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:13.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:13.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:13.390 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:11:13.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:11:13.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:13.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:13.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:14.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:14.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:14.812 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:11:14.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:11:14.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:14.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:15.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:16.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:16.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:16.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:11:16.291 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:11:16.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:16.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:16.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:17.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:17.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:17.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:11:17.924 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:11:17.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:17.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:18.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:19.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:19.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:19.925 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:11:19.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:11:19.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:19.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:20.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:21.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:21.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:21.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:11:21.727 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:11:21.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:21.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:22.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:23.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:23.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:23.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:11:23.529 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:11:23.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:23.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:24.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:25.251 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:11:25.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:11:25.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:25.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:27.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:27.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:27.346 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:11:27.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:11:27.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:27.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:27.848 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3263024197,v1:127.0.0.1:6811/3263024197] [v2:127.0.0.1:6812/3263024197,v1:127.0.0.1:6813/3263024197] exists,up 6bd8c733-af70-4894-adc8-d566b8f480e7 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:11:27.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:11:27.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-04-25T07:11:27.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:11:27.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=804177d8-31aa-42c8-8c44-731e1842ae09 2024-04-25T07:11:27.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 804177d8-31aa-42c8-8c44-731e1842ae09' 2024-04-25T07:11:27.856 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 804177d8-31aa-42c8-8c44-731e1842ae09 2024-04-25T07:11:27.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:11:27.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfAipmbhdoNBAAC7fTUVQckb4Kpqz8DjRbrA== 2024-04-25T07:11:27.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfAipmbhdoNBAAC7fTUVQckb4Kpqz8DjRbrA=="}' 2024-04-25T07:11:27.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 804177d8-31aa-42c8-8c44-731e1842ae09 -i td/ec-error-rollforward/2/new.json 2024-04-25T07:11:28.384 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:11:28.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-04-25T07:11:28.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfAipmbhdoNBAAC7fTUVQckb4Kpqz8DjRbrA== --osd-uuid 804177d8-31aa-42c8-8c44-731e1842ae09 2024-04-25T07:11:28.433 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:28.434+0000 7fa3370247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:28.522 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:28.522+0000 7fa3370247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:28.524 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:28.526+0000 7fa3370247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:28.524 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:28.526+0000 7fa3370247c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:11:28.524 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:28.526+0000 7fa3370247c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:11:28.524 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:28.526+0000 7fa3370247c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:11:28.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:28.626+0000 7fa3370247c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-04-25T07:11:34.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-04-25T07:11:34.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:11:34.403 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:11:34.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:11:34.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:11:34.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:11:34.952 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:11:34.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:11:34.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:11:34.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:11:34.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:11:34.982 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:34.982+0000 7f20e83817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:34.990 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:34.990+0000 7f20e83817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:34.990+0000 7f20e83817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:35.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:11:35.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:35.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:11:35.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:35.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:35.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:35.378 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:11:35.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:35.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:35.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:35.549 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:35.550+0000 7f20e83817c0 -1 Falling back to public interface 2024-04-25T07:11:35.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:36.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:36.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:36.817 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:11:36.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:11:36.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:36.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:37.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:38.064 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:11:38.066+0000 7f20e83817c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:11:38.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:38.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:38.234 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:11:38.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:11:38.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:38.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:38.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:39.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:39.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:39.794 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:11:39.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:11:39.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:39.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:40.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:41.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:41.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:41.219 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:11:41.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:11:41.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:41.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:41.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:42.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:42.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:42.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:11:42.642 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:11:42.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:42.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:43.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:44.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:44.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:44.111 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:11:44.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:11:44.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:44.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:44.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:45.569 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:11:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:11:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:46.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:47.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:47.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:47.133 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:11:47.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:11:47.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:47.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:47.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:48.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:48.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:48.587 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:11:48.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:11:48.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:48.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:49.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:50.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:50.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:50.052 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:11:50.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:11:50.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:50.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:50.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:51.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:51.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:51.747 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:11:51.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:11:51.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:51.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:52.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:53.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:53.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:53.407 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:11:53.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:11:53.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:53.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:54.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:55.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:55.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:55.110 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:11:55.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:11:55.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:55.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:55.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:56.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:56.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:56.922 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:11:56.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:11:56.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:56.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:58.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:58.559 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:11:58.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:58.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:11:58.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:58.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:59.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:00.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:00.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:00.371 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:12:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:12:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:00.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:12:00.805 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1116151506,v1:127.0.0.1:6819/1116151506] [v2:127.0.0.1:6820/1116151506,v1:127.0.0.1:6821/1116151506] exists,up 804177d8-31aa-42c8-8c44-731e1842ae09 2024-04-25T07:12:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-04-25T07:12:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:12:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:12:00.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:12:00.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:12:00.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-04-25T07:12:00.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:12:00.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:00.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:00.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:00.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-04-25T07:12:00.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-04-25T07:12:00.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:00.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:12:00.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:12:00.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:12:00.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:00.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:00.817 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:00.817 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:00.817 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:12:00.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:00.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:00.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:00.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:12:00.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:12:00.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:12:00.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:12:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:12:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:12:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-04-25T07:12:00.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:12:00.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=10f1f4c9-9862-4e53-8500-1c1fc6e90625 2024-04-25T07:12:00.820 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 10f1f4c9-9862-4e53-8500-1c1fc6e90625 2024-04-25T07:12:00.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 10f1f4c9-9862-4e53-8500-1c1fc6e90625' 2024-04-25T07:12:00.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:12:00.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAAipmIBWGMRAA3fnKWliCKqZqggcWLzZ12w== 2024-04-25T07:12:00.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAAipmIBWGMRAA3fnKWliCKqZqggcWLzZ12w=="}' 2024-04-25T07:12:00.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 10f1f4c9-9862-4e53-8500-1c1fc6e90625 -i td/ec-error-rollforward/3/new.json 2024-04-25T07:12:01.275 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:12:01.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-04-25T07:12:01.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAAipmIBWGMRAA3fnKWliCKqZqggcWLzZ12w== --osd-uuid 10f1f4c9-9862-4e53-8500-1c1fc6e90625 2024-04-25T07:12:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:01.318+0000 7f1d4486e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:01.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:01.334+0000 7f1d4486e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:01.335 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:01.334+0000 7f1d4486e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:01.337 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:01.338+0000 7f1d4486e7c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:12:01.337 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:01.338+0000 7f1d4486e7c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:12:01.337 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:01.338+0000 7f1d4486e7c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:12:01.373 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:01.374+0000 7f1d4486e7c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-04-25T07:12:07.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-04-25T07:12:07.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:12:07.249 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T07:12:07.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:12:07.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:12:07.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:12:07.796 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T07:12:07.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:12:07.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:07.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:12:07.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:12:07.827 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:07.826+0000 7fa0ade377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:07.862 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:07.862+0000 7fa0ade377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:07.864 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:07.866+0000 7fa0ade377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:08.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:12:08.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:08.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:12:08.221 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:12:08.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:08.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:08.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:08.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:08.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:08.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:08.430 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:08.430+0000 7fa0ade377c0 -1 Falling back to public interface 2024-04-25T07:12:08.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:09.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:09.619 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:12:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:12:09.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:09.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:10.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:11.011 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:11.010+0000 7fa0ade377c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:12:11.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:11.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:11.071 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:12:11.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:12:11.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:11.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:11.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:12.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:12.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:12.701 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:12:12.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:12:12.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:12.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:13.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:14.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:14.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:14.225 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:12:14.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:12:14.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:14.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:14.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:15.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:15.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:15.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:12:15.635 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:12:15.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:15.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:16.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:17.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:17.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:17.083 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:12:17.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:12:17.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:17.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:17.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:18.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:18.509 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:12:18.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:12:18.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:18.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:18.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:19.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:19.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:19.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:12:19.911 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:12:19.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:19.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:20.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:21.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:21.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:21.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:12:21.323 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:12:21.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:21.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:21.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:22.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:22.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:22.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:12:22.807 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:12:22.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:22.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:23.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:24.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:24.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:24.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:12:24.440 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:12:24.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:24.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:25.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:26.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:26.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:26.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:12:26.382 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:12:26.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:26.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:27.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:28.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:28.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:28.182 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:12:28.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:12:28.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:28.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:28.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:29.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:29.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:29.621 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:12:29.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:12:29.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:29.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:30.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:31.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:31.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:31.413 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:12:31.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:12:31.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:31.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:32.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:33.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:33.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:33.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:12:33.164 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:12:33.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:33.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:33.656 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3426298314,v1:127.0.0.1:6827/3426298314] [v2:127.0.0.1:6828/3426298314,v1:127.0.0.1:6829/3426298314] exists,up 10f1f4c9-9862-4e53-8500-1c1fc6e90625 2024-04-25T07:12:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-04-25T07:12:34.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-04-25T07:12:35.112 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'ec' already exists 2024-04-25T07:12:35.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-04-25T07:12:37.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-04-25T07:12:37.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 87277 2024-04-25T07:12:37.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=89937 2024-04-25T07:12:37.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-04-25T07:12:37.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-04-25T07:12:38.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 89958' 2024-04-25T07:12:38.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-04-25T07:12:38.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 89959' 2024-04-25T07:12:38.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-04-25T07:12:38.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 89960' 2024-04-25T07:12:38.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-04-25T07:12:38.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-04-25T07:12:39.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-04-25T07:12:39.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:12:39.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:12:39.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:12:39.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:12:39.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:12:39.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:12:39.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 89937 89958 89959 89960 2024-04-25T07:12:39.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-04-25T07:12:39.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-04-25T07:12:39.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:12:39.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:12:39.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:12:39.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:12:39.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:39.494 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:12:39.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:12:39.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-04-25T07:12:39.496 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:12:39.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:12:39.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:39.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-04-25T07:12:39.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:12:39.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:12:39.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:12:39.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:12:39.524 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:39.529+0000 7f7a49b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:39.527 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:39.533+0000 7f7a49b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:39.529 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:39.533+0000 7f7a49b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:39.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:12:39.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:39.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:12:39.909 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:12:39.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:39.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:39.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:39.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:39.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:39.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:40.085 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:40.089+0000 7f7a49b177c0 -1 Falling back to public interface 2024-04-25T07:12:40.360 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1038428475,v1:127.0.0.1:6803/1038428475] [v2:127.0.0.1:6804/1038428475,v1:127.0.0.1:6805/1038428475] exists,up 57d13c1b-e6f1-4fcf-9598-57d0d0cf436b 2024-04-25T07:12:40.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:40.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:12:40.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:40.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:40.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-04-25T07:12:40.366 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:12:40.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:12:40.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:40.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-04-25T07:12:40.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:12:40.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:12:40.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:12:40.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:12:40.391 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:40.393+0000 7f9f9596a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:40.412 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:40.417+0000 7f9f9596a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:40.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:40.417+0000 7f9f9596a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:12:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:12:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:40.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:40.782 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:12:40.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:40.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:40.958 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:40.961+0000 7f9f9596a7c0 -1 Falling back to public interface 2024-04-25T07:12:41.198 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3263024197,v1:127.0.0.1:6811/3263024197] [v2:127.0.0.1:6812/3263024197,v1:127.0.0.1:6813/3263024197] exists,up 6bd8c733-af70-4894-adc8-d566b8f480e7 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-04-25T07:12:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:41.202 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:12:41.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-04-25T07:12:41.204 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:12:41.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:12:41.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:41.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-04-25T07:12:41.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:12:41.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:12:41.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:12:41.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:12:41.225 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:41.229+0000 7f4065e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:41.227 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:41.233+0000 7f4065e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:41.228 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:41.233+0000 7f4065e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:41.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:12:41.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:41.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:12:41.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:41.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:41.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:41.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:41.628 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:12:41.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:41.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:12:41.786 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:41.789+0000 7f4065e8b7c0 -1 Falling back to public interface 2024-04-25T07:12:42.038 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1116151506,v1:127.0.0.1:6819/1116151506] [v2:127.0.0.1:6820/1116151506,v1:127.0.0.1:6821/1116151506] exists,up 804177d8-31aa-42c8-8c44-731e1842ae09 2024-04-25T07:12:42.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:42.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:42.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:42.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-04-25T07:12:42.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:12:42.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:12:42.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:12:42.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:12:42.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-04-25T07:12:42.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:12:42.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:42.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:42.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:42.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-04-25T07:12:42.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-04-25T07:12:42.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:42.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:12:42.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:12:42.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:12:42.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:42.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:42.046 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:42.047 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:42.047 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:12:42.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:42.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' 2024-04-25T07:12:42.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:42.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:12:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:12:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:12:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:12:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-04-25T07:12:42.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:12:42.049 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T07:12:42.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=98315754-77b2-4a9f-bf0e-ca8d1a4e0277 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.82769/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:42.051 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-04-25T07:12:42.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:12:42.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:12:42.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:12:42.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:12:42.067 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:42.073+0000 7f61a6ab27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:42.075 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:42.081+0000 7f61a6ab27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:42.076 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:42.081+0000 7f61a6ab27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:42.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:12:42.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:42.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:12:42.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:42.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:42.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:42.459 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:12:42.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:42.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:42.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:12:42.634 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:42.637+0000 7f61a6ab27c0 -1 Falling back to public interface 2024-04-25T07:12:42.990 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3426298314,v1:127.0.0.1:6827/3426298314] [v2:127.0.0.1:6828/3426298314,v1:127.0.0.1:6829/3426298314] exists,up 10f1f4c9-9862-4e53-8500-1c1fc6e90625 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:12:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:12:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:12:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:12:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:12:43.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:12:43.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:12:43.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:12:43.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:12:43.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:12:43.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:12:43.499 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:43.501+0000 7f7a49b177c0 -1 osd.0 26 log_to_monitors true 2024-04-25T07:12:43.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:12:43.506 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:12:43.506 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:12:43.506 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T07:12:43.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:12:43.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:43.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:12:44.782 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:44.785+0000 7f9f9596a7c0 -1 osd.1 26 log_to_monitors true 2024-04-25T07:12:45.822 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:45.825+0000 7f4065e8b7c0 -1 osd.2 25 log_to_monitors true 2024-04-25T07:12:46.427 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:46.429+0000 7f61a6ab27c0 -1 osd.3 26 log_to_monitors true 2024-04-25T07:12:48.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051587 2024-04-25T07:12:48.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051587 2024-04-25T07:12:48.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587' 2024-04-25T07:12:48.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:48.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:12:48.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953475 2024-04-25T07:12:48.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953475 2024-04-25T07:12:48.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-04-25T07:12:48.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:48.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:12:49.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789955 2024-04-25T07:12:49.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789955 2024-04-25T07:12:49.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-158913789955' 2024-04-25T07:12:49.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:12:49.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:12:49.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=167503724547 2024-04-25T07:12:49.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 167503724547 2024-04-25T07:12:49.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-158913789955 3-167503724547' 2024-04-25T07:12:49.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:12:49.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051587 2024-04-25T07:12:49.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:12:49.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:12:49.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051587 2024-04-25T07:12:49.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:12:49.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051587 2024-04-25T07:12:49.250 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 124554051587 2024-04-25T07:12:49.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-04-25T07:12:49.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:12:49.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 124554051587 2024-04-25T07:12:49.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:12:50.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:12:50.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:12:51.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051588 -lt 124554051587 2024-04-25T07:12:51.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:12:51.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-137438953475 2024-04-25T07:12:51.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:12:51.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:12:51.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:12:51.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-137438953475 2024-04-25T07:12:51.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953475 2024-04-25T07:12:51.066 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 137438953475 2024-04-25T07:12:51.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-04-25T07:12:51.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:12:51.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953475 -lt 137438953475 2024-04-25T07:12:51.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:12:51.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-158913789955 2024-04-25T07:12:51.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:12:51.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:12:51.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-158913789955 2024-04-25T07:12:51.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:12:51.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789955 2024-04-25T07:12:51.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2024-04-25T07:12:51.497 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 158913789955 2024-04-25T07:12:51.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:12:51.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789955 -lt 158913789955 2024-04-25T07:12:51.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:12:51.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-167503724547 2024-04-25T07:12:51.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:12:51.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:12:51.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:12:51.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-167503724547 2024-04-25T07:12:51.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=167503724547 2024-04-25T07:12:51.935 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 167503724547 2024-04-25T07:12:51.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 167503724547' 2024-04-25T07:12:51.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:12:52.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 167503724547 -lt 167503724547 2024-04-25T07:12:52.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:12:52.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:52.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:52.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:12:52.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:12:52.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:12:52.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:12:52.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:12:52.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:12:52.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:12:52.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:12:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:12:53.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:12:53.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:12:53.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:12:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:12:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:12:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:12:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-04-25T07:12:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:12:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:12:53.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:12:53.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:12:53.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:12:53.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:12:53.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:12:53.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:12:54.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:12:54.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:12:54.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:12:54.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:12:54.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:12:54.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:12:54.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:12:54.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:54.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:12:54.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:12:54.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:54.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:12:54.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:12:54.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:12:54.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:12:54.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:54.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:12:54.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.82769 2024-04-25T07:12:54.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:12:54.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:12:54.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:12:54.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/ec-error-rollforward 0 2024-04-25T07:12:54.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:12:54.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:12:54.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:12:54.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:12:54.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:12:54.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:12:54.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:12:54.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:12:54.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:12:54.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:12:54.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:12:54.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:12:54.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:12:54.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:12:54.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:12:54.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:54.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:12:54.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:12:54.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:54.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:12:54.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:12:54.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:12:54.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:12:54.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:54.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.82769 2024-04-25T07:12:54.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.82769 2024-04-25T07:12:54.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:12:54.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:12:54.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:12:54.239 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:12:54.239 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:12:54.294 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-04-25T07:12:54.294 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-04-25T07:12:54.347 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-bench 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-04-25T07:12:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-04-25T07:12:54.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-04-25T07:12:54.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-04-25T07:12:54.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=7f6821d6-1f81-4074-ae7e-f6ba186af44a --auth-supported=none ' 2024-04-25T07:12:54.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-04-25T07:12:54.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-04-25T07:12:54.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-04-25T07:12:54.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:12:54.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:12:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:12:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:12:54.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:12:54.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:12:54.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:12:54.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:12:54.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:12:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:12:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:12:54.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:12:54.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:54.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:12:54.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:12:54.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:54.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:12:54.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:12:54.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:12:54.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:12:54.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:54.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:12:54.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92468 2024-04-25T07:12:54.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:12:54.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:12:54.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-04-25T07:12:54.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:12:54.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:54.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:12:54.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.92468 2024-04-25T07:12:54.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:12:54.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:12:54.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-04-25T07:12:54.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-04-25T07:12:54.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:12:54.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:54.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:54.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:54.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:54.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:12:54.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92468/$cluster-$name.asok' 2024-04-25T07:12:54.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.92468/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:12:55.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:12:55.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:12:55.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:12:55.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:12:55.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:12:55.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:12:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:12:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:12:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:12:55.039 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:12:55.039 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:55.039 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:12:55.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92468/ceph-mon.a.asok 2024-04-25T07:12:55.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:12:55.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.92468/ceph-mon.a.asok config get fsid 2024-04-25T07:12:55.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:12:55.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:12:55.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:12:55.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:12:55.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:12:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:12:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:12:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:12:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:12:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:12:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92468/ceph-mon.a.asok 2024-04-25T07:12:55.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:12:55.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.92468/ceph-mon.a.asok config get mon_host 2024-04-25T07:12:55.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-04-25T07:12:55.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-04-25T07:12:55.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:12:55.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:12:55.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:12:55.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-04-25T07:12:55.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:12:56.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:12:56.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:56.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:56.141 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:56.141 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:56.141 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:12:56.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92468/$cluster-$name.asok' 2024-04-25T07:12:56.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:12:56.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.92468/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:12:56.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-04-25T07:12:56.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7f6821d6-1f81-4074-ae7e-f6ba186af44a --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-04-25T07:12:56.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92468/$cluster-$name.asok' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92468/$cluster-$name.asok' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-04-25T07:12:56.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:56.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:56.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:56.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:56.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:12:56.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:12:56.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-04-25T07:12:56.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:12:56.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=536cd0dc-291d-4fd0-af0b-09f624944b0e 2024-04-25T07:12:56.176 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 536cd0dc-291d-4fd0-af0b-09f624944b0e 2024-04-25T07:12:56.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 536cd0dc-291d-4fd0-af0b-09f624944b0e' 2024-04-25T07:12:56.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:12:56.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4AipmSSapCxAATbgNtjN222ac3tKgDqQ+ow== 2024-04-25T07:12:56.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4AipmSSapCxAATbgNtjN222ac3tKgDqQ+ow=="}' 2024-04-25T07:12:56.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 536cd0dc-291d-4fd0-af0b-09f624944b0e -i td/osd-bench/0/new.json 2024-04-25T07:12:56.457 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:12:56.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-04-25T07:12:56.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7f6821d6-1f81-4074-ae7e-f6ba186af44a --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.92468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4AipmSSapCxAATbgNtjN222ac3tKgDqQ+ow== --osd-uuid 536cd0dc-291d-4fd0-af0b-09f624944b0e 2024-04-25T07:12:56.503 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:56.509+0000 7fa47935e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:56.517 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:56.521+0000 7fa47935e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:56.518 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:56.521+0000 7fa47935e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:56.519 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:56.521+0000 7fa47935e7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:12:56.519 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:56.521+0000 7fa47935e7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:12:56.519 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:56.521+0000 7fa47935e7c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:12:56.557 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:12:56.561+0000 7fa47935e7c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-04-25T07:13:02.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-04-25T07:13:02.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:13:02.479 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:13:02.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:13:02.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:13:03.021 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:13:03.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:13:03.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7f6821d6-1f81-4074-ae7e-f6ba186af44a --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.92468/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:13:03.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:13:03.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:13:03.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:13:03.051 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:13:03.053+0000 7fdba81cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:03.057 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:13:03.061+0000 7fdba81cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:03.058 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:13:03.061+0000 7fdba81cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:03.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:13:03.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:03.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:13:03.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:03.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:03.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:03.486 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:13:03.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:03.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:03.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:03.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:13:03.641+0000 7fdba81cb7c0 -1 Falling back to public interface 2024-04-25T07:13:03.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:04.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:04.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:04.928 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:13:04.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:13:04.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:04.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:05.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:06.088 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:13:06.093+0000 7fdba81cb7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:13:06.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:06.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:06.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:13:06.350 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:13:06.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:06.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:06.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:07.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:07.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:07.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:13:07.926 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:13:07.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:07.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:08.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:09.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:09.375 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:13:09.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:09.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:13:09.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:09.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:09.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:10.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:10.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:10.835 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:13:10.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:13:10.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:10.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:11.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:12.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:12.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:12.279 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:13:12.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:13:12.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:12.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:12.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:13.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:13.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:13.687 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:13:13.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:13:13.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:13.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:14.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:15.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:15.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:15.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:13:15.139 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:13:15.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:15.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:15.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:16.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:16.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:16.591 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:13:16.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:13:16.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:16.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:17.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:18.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:18.138 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:13:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:13:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:18.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:18.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:19.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:19.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:19.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:13:19.758 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:13:19.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:19.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:20.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:21.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:21.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:21.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:13:21.795 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:13:21.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:21.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:22.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:23.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:23.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:23.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:13:23.613 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:13:23.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:23.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:24.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:25.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:25.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:25.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:13:25.546 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:13:25.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:25.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:26.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:27.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:27.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:27.257 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:13:27.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:13:27.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:27.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:27.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:28.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:28.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:28.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:13:28.974 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:13:28.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:28.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:13:29.424 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/920806602,v1:127.0.0.1:6803/920806602] [v2:127.0.0.1:6804/920806602,v1:127.0.0.1:6805/920806602] exists,up 536cd0dc-291d-4fd0-af0b-09f624944b0e 2024-04-25T07:13:29.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:13:29.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:13:29.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:13:29.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-04-25T07:13:29.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-04-25T07:13:29.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-04-25T07:13:29.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-04-25T07:13:29.443 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-04-25T07:13:29.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-04-25T07:13:29.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-04-25T07:13:29.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-04-25T07:13:29.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-04-25T07:13:29.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-04-25T07:13:29.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-04-25T07:13:29.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-04-25T07:13:29.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-04-25T07:13:29.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:13:29.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:13:29.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=osd_bench_max_block_size 2024-04-25T07:13:29.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:13:29.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:13:29.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-04-25T07:13:29.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:13:29.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:13:29.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-04-25T07:13:29.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:13:29.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-04-25T07:13:29.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-04-25T07:13:29.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-04-25T07:13:29.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:13:29.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:13:29.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=bench_small_size_max_iops 2024-04-25T07:13:29.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:13:29.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:13:29.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-04-25T07:13:29.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:13:29.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:13:29.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:13:29.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-04-25T07:13:29.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:13:29.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:13:29.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-04-25T07:13:29.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:13:29.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-04-25T07:14:17.669 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:14:17.670 INFO:tasks.workunit.client.0.smithi088.stdout: "bytes_written": 1073741824, 2024-04-25T07:14:17.670 INFO:tasks.workunit.client.0.smithi088.stdout: "blocksize": 4194304, 2024-04-25T07:14:17.670 INFO:tasks.workunit.client.0.smithi088.stdout: "elapsed_sec": 23.28914305, 2024-04-25T07:14:17.670 INFO:tasks.workunit.client.0.smithi088.stdout: "bytes_per_sec": 46104823.251536511, 2024-04-25T07:14:17.670 INFO:tasks.workunit.client.0.smithi088.stdout: "iops": 10.992246449360016 2024-04-25T07:14:17.670 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:14:17.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-04-25T07:14:18.153 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:14:18.154 INFO:tasks.workunit.client.0.smithi088.stdout: "bytes_written": 10, 2024-04-25T07:14:18.154 INFO:tasks.workunit.client.0.smithi088.stdout: "blocksize": 14456, 2024-04-25T07:14:18.154 INFO:tasks.workunit.client.0.smithi088.stdout: "elapsed_sec": 0.058503702999999997, 2024-04-25T07:14:18.154 INFO:tasks.workunit.client.0.smithi088.stdout: "bytes_per_sec": 170.92935125832975, 2024-04-25T07:14:18.154 INFO:tasks.workunit.client.0.smithi088.stdout: "iops": 0.011824111182784293 2024-04-25T07:14:18.154 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:14:18.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-04-25T07:14:18.496 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:14:18.496 INFO:tasks.workunit.client.0.smithi088.stdout: "bytes_written": 1, 2024-04-25T07:14:18.496 INFO:tasks.workunit.client.0.smithi088.stdout: "blocksize": 14456, 2024-04-25T07:14:18.496 INFO:tasks.workunit.client.0.smithi088.stdout: "elapsed_sec": 0.038903502999999999, 2024-04-25T07:14:18.496 INFO:tasks.workunit.client.0.smithi088.stdout: "bytes_per_sec": 25.70462613610913, 2024-04-25T07:14:18.496 INFO:tasks.workunit.client.0.smithi088.stdout: "iops": 0.0017781285373622807 2024-04-25T07:14:18.496 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:14:18.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-04-25T07:14:18.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:14:18.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:14:18.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:14:18.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:14:18.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:14:18.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:14:18.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:14:18.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:14:18.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:14:18.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:14:18.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:14:18.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:14:18.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:14:18.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:14:18.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:14:18.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:18.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:14:18.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:14:18.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:18.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:14:18.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:14:18.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:14:18.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:14:18.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:18.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:14:18.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92468 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-bench 0 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:14:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:14:18.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:14:18.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:14:18.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:14:18.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:14:18.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:14:18.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:14:18.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:14:18.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:18.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:14:18.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:14:18.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:18.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:14:18.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:14:18.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:14:18.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:14:18.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:18.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92468 2024-04-25T07:14:18.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92468 2024-04-25T07:14:18.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:14:18.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:14:18.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:14:18.896 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:14:18.896 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:14:18.952 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-04-25T07:14:18.952 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-04-25T07:14:19.007 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:14:19.014 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:14:19.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-bluefs-volume-ops 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:14:19.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-04-25T07:14:19.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-04-25T07:14:19.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:14:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-04-25T07:14:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_bluestore2 2024-04-25T07:14:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_bluestore_expand' 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:14:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:14:19.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:14:19.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:14:19.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:14:19.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:14:19.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:14:19.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:14:19.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:14:19.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:19.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:14:19.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:14:19.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:14:19.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:14:19.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:14:19.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:14:19.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:14:19.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:19.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:14:19.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94617 2024-04-25T07:14:19.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:14:19.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:14:19.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:14:19.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:14:19.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:19.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:14:19.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94617 2024-04-25T07:14:19.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:14:19.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:14:19.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:14:19.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:14:19.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:14:19.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-04-25T07:14:19.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:14:19.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-04-25T07:14:19.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-04-25T07:14:19.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-04-25T07:14:19.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:14:19.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-04-25T07:14:19.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-04-25T07:14:19.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none ' 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:14:19.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:14:19.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:14:19.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:14:19.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:14:19.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:14:19.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:14:19.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:19.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:19.499 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:19.499 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:19.499 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:14:19.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:14:19.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:14:19.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:14:19.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:14:19.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:14:19.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:14:19.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:14:19.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:14:19.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:14:19.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:14:19.695 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:14:19.696 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:19.696 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:19.696 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:14:19.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94617/ceph-mon.a.asok 2024-04-25T07:14:19.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:14:19.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94617/ceph-mon.a.asok config get fsid 2024-04-25T07:14:19.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:14:19.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:14:19.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:14:19.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:14:19.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:14:19.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:14:19.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:14:19.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:14:19.813 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:19.813 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:19.813 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:14:19.813 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94617/ceph-mon.a.asok 2024-04-25T07:14:19.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:14:19.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94617/ceph-mon.a.asok config get mon_host 2024-04-25T07:14:19.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:14:19.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:14:19.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:14:19.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:14:19.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:14:19.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:14:19.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:14:20.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:14:20.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:20.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:20.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:20.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:20.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:14:20.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:14:20.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:14:20.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:14:20.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:14:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:14:20.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:14:20.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:14:20.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=85e4cd7e-efc8-49c2-a5ee-9fe41f509b63 2024-04-25T07:14:20.842 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 85e4cd7e-efc8-49c2-a5ee-9fe41f509b63 2024-04-25T07:14:20.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 85e4cd7e-efc8-49c2-a5ee-9fe41f509b63' 2024-04-25T07:14:20.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:14:20.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMAipmT0RmMxAAcmBdHgb2bt+jBFSxmO/T5w== 2024-04-25T07:14:20.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMAipmT0RmMxAAcmBdHgb2bt+jBFSxmO/T5w=="}' 2024-04-25T07:14:20.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 85e4cd7e-efc8-49c2-a5ee-9fe41f509b63 -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:14:21.146 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:14:21.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:14:21.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMAipmT0RmMxAAcmBdHgb2bt+jBFSxmO/T5w== --osd-uuid 85e4cd7e-efc8-49c2-a5ee-9fe41f509b63 2024-04-25T07:14:21.193 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:21.195+0000 7f23da4237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:21.214 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:21.215+0000 7f23da4237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:21.216 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:21.219+0000 7f23da4237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:21.216 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:21.219+0000 7f23da4237c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:14:21.216 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:21.219+0000 7f23da4237c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:14:21.216 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:21.219+0000 7f23da4237c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:14:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:21.255+0000 7f23da4237c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:14:30.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:14:30.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:14:30.584 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:14:30.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:14:30.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:14:31.129 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:14:31.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:14:31.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:14:31.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:14:31.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:14:31.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:14:31.154 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:31.155+0000 7fed584bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:31.156 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:31.159+0000 7fed584bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:31.157 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:31.159+0000 7fed584bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:14:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:14:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:14:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:14:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:14:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:31.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:14:31.538 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:14:31.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:31.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:32.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:32.206 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:32.207+0000 7fed584bb7c0 -1 Falling back to public interface 2024-04-25T07:14:33.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:33.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:33.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:14:33.025 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:14:33.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:33.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:33.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:34.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:34.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:34.441 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:14:34.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:14:34.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:34.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:34.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:35.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:35.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:35.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:14:35.896 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:14:35.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:35.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:36.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:37.005 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:14:37.007+0000 7fed584bb7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:14:37.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:37.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:37.370 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:14:37.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:14:37.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:37.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:37.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:38.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:38.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:38.790 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:14:38.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:14:38.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:38.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:39.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:40.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:40.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:40.206 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:14:40.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:14:40.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:40.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:40.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:41.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:41.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:41.611 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:14:41.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:14:41.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:41.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:42.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:43.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:43.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:43.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:14:43.098 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:14:43.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:43.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:43.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:44.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:44.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:44.519 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:14:44.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:14:44.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:44.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:45.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:46.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:46.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:46.086 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:14:46.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:14:46.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:46.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:46.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:47.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:47.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:47.591 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:14:47.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:14:47.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:47.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:48.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:49.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:49.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:49.180 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:14:49.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:14:49.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:49.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:49.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:50.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:50.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:50.728 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:14:50.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:14:50.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:50.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:51.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:52.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:52.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:52.470 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:14:52.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:14:52.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:52.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:53.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:54.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:54.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:54.183 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:14:54.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:14:54.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:54.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:55.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:56.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:56.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:56.238 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:14:56.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:14:56.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:56.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:57.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:58.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:58.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:58.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:14:58.289 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:14:58.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:58.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:59.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:00.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:00.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:00.103 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T07:15:00.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:15:00.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:00.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:15:00.555 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3723262312,v1:127.0.0.1:6803/3723262312] [v2:127.0.0.1:6804/3723262312,v1:127.0.0.1:6805/3723262312] exists,up 85e4cd7e-efc8-49c2-a5ee-9fe41f509b63 2024-04-25T07:15:00.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:00.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:00.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:00.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:15:00.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=96078 2024-04-25T07:15:00.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:15:00.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:15:00.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:15:00.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:15:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:15:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:00.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:15:00.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:15:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1dd411e3-2838-4aa5-96cc-e48db2d17c8a 2024-04-25T07:15:00.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1dd411e3-2838-4aa5-96cc-e48db2d17c8a' 2024-04-25T07:15:00.564 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 1dd411e3-2838-4aa5-96cc-e48db2d17c8a 2024-04-25T07:15:00.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:15:00.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0AipmjfssIxAA5w/jdR6ENCuU/D0UaHmFWw== 2024-04-25T07:15:00.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0AipmjfssIxAA5w/jdR6ENCuU/D0UaHmFWw=="}' 2024-04-25T07:15:00.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1dd411e3-2838-4aa5-96cc-e48db2d17c8a -i td/osd-bluefs-volume-ops/1/new.json 2024-04-25T07:15:01.070 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:15:01.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-04-25T07:15:01.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0AipmjfssIxAA5w/jdR6ENCuU/D0UaHmFWw== --osd-uuid 1dd411e3-2838-4aa5-96cc-e48db2d17c8a 2024-04-25T07:15:01.122 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:01.122+0000 7f82581a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:01.124 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:01.126+0000 7f82581a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:01.125 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:01.126+0000 7f82581a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:01.126 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:01.126+0000 7f82581a67c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:15:01.126 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:01.126+0000 7f82581a67c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:15:01.126 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:01.126+0000 7f82581a67c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:15:01.170 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:01.170+0000 7f82581a67c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-04-25T07:15:11.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-04-25T07:15:11.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:15:11.009 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:15:11.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:15:11.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:15:11.551 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:15:11.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:15:11.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:15:11.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:15:11.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:15:11.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:15:11.581 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:11.582+0000 7f6622e997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:11.600 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:11.602+0000 7f6622e997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:11.601 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:11.602+0000 7f6622e997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:11.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:15:11.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:15:11.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:15:11.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:15:11.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:15:11.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:11.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:15:11.955 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:15:11.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:11.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:12.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:12.654 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:12.654+0000 7f6622e997c0 -1 Falling back to public interface 2024-04-25T07:15:13.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:13.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:13.415 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:15:13.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:15:13.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:13.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:13.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:14.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:14.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:14.874 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:15:14.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:15:14.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:14.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:15.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:16.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:16.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:16.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:15:16.276 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:15:16.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:16.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:16.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:17.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:17.454+0000 7f6622e997c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:15:17.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:17.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:17.705 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:15:17.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:15:17.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:17.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:18.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:19.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:19.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:19.287 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:15:19.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:15:19.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:19.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:19.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:20.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:20.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:20.786 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:15:20.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:15:20.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:20.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:21.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:15:22.246 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:15:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:22.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:22.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:23.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:23.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:23.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:15:23.836 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:15:23.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:23.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:24.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:25.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:25.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:25.396 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:15:25.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:15:25.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:25.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:25.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:26.825 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:15:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:15:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:27.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:28.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:28.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:28.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:15:28.282 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:15:28.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:28.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:28.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:29.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:29.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:29.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:15:29.770 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:15:29.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:29.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:30.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:31.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:31.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:31.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:15:31.388 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:15:31.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:31.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:32.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:33.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:33.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:15:33.078 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:15:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:33.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:34.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:34.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:34.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:15:34.854 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:15:34.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:34.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:35.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:36.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:36.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:36.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:15:36.503 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:15:36.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:36.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:37.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:38.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:38.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:38.370 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:15:38.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:15:38.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:38.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:39.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:40.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:40.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:40.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:15:40.048 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T07:15:40.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:40.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:40.493 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/675147512,v1:127.0.0.1:6811/675147512] [v2:127.0.0.1:6812/675147512,v1:127.0.0.1:6813/675147512] exists,up 1dd411e3-2838-4aa5-96cc-e48db2d17c8a 2024-04-25T07:15:40.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:40.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:40.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:40.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=98597 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:15:40.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:15:40.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:15:40.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:15:40.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:15:40.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:15:40.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:15:40.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:15:40.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:15:40.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6' 2024-04-25T07:15:40.501 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6 2024-04-25T07:15:40.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:15:40.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcAypm99EMHxAAzqA1MbkX74IDWoTbHHw2vw== 2024-04-25T07:15:40.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcAypm99EMHxAAzqA1MbkX74IDWoTbHHw2vw=="}' 2024-04-25T07:15:40.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6 -i td/osd-bluefs-volume-ops/2/new.json 2024-04-25T07:15:41.046 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:15:41.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-04-25T07:15:41.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcAypm99EMHxAAzqA1MbkX74IDWoTbHHw2vw== --osd-uuid ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6 2024-04-25T07:15:41.092 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:41.097+0000 7fb6a23f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:41.100 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:41.101+0000 7fb6a23f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:41.101 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:41.105+0000 7fb6a23f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:41.101 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:41.105+0000 7fb6a23f87c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:15:41.101 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:41.105+0000 7fb6a23f87c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:15:41.101 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:41.105+0000 7fb6a23f87c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:15:41.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:41.141+0000 7fb6a23f87c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-04-25T07:15:51.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-04-25T07:15:51.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:15:51.330 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:15:51.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:15:51.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:15:51.885 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:15:51.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:15:51.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:15:51.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:15:51.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:15:51.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:15:51.916 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:51.921+0000 7f14735427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:51.918 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:51.921+0000 7f14735427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:51.920 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:51.921+0000 7f14735427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:52.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:15:52.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:15:52.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:15:52.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:15:52.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:15:52.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:52.281 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:15:52.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:15:52.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:52.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:52.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:52.963 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:52.965+0000 7f14735427c0 -1 Falling back to public interface 2024-04-25T07:15:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:53.741 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:15:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:15:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:54.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:55.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:55.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:55.163 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:15:55.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:15:55.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:55.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:55.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:56.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:56.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:56.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:15:56.604 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:15:56.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:56.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:57.810 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:15:57.813+0000 7f14735427c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:15:58.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:58.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:58.041 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:15:58.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:15:58.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:58.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:58.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:59.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:59.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:59.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:15:59.607 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:15:59.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:59.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:00.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:01.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:16:01.019 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:16:01.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:01.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:01.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:02.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:02.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:02.444 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:16:02.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:16:02.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:02.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:02.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:03.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:03.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:03.878 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:16:03.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:16:03.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:03.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:04.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:05.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:05.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:05.290 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:16:05.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:16:05.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:05.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:05.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:06.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:06.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:06.782 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:16:06.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:16:06.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:06.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:07.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:08.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:08.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:08.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:16:08.214 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:16:08.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:08.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:08.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:09.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:09.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:09.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:16:09.763 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:16:09.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:09.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:10.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:11.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:11.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:11.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:16:11.251 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:16:11.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:11.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:12.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:13.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:13.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:13.519 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:16:13.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:16:13.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:13.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:14.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:15.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:15.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:15.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:16:15.585 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:16:15.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:15.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:16.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:17.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:17.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:17.758 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:16:17.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:16:17.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:17.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:18.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:19.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:19.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:19.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:16:19.721 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:16:19.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:19.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:20.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:21.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:21.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:21.865 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T07:16:21.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:16:21.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:21.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:22.293 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/272115526,v1:127.0.0.1:6819/272115526] [v2:127.0.0.1:6820/272115526,v1:127.0.0.1:6821/272115526] exists,up ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6 2024-04-25T07:16:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:16:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:16:22.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:16:22.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:16:22.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=101115 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:16:22.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:22.299 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:16:22.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:16:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:16:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:16:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:16:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:16:22.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:16:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5 2024-04-25T07:16:22.302 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5 2024-04-25T07:16:22.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5' 2024-04-25T07:16:22.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:16:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGAypm17ptExAA9bOGfTRs7v6YuonCxgg0sg== 2024-04-25T07:16:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGAypm17ptExAA9bOGfTRs7v6YuonCxgg0sg=="}' 2024-04-25T07:16:22.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5 -i td/osd-bluefs-volume-ops/3/new.json 2024-04-25T07:16:22.786 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:16:22.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-04-25T07:16:22.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGAypm17ptExAA9bOGfTRs7v6YuonCxgg0sg== --osd-uuid d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5 2024-04-25T07:16:22.829 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:22.832+0000 7f5fdbbeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:22.872 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:22.872+0000 7f5fdbbeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:22.873 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:22.876+0000 7f5fdbbeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:22.874 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:22.876+0000 7f5fdbbeb7c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:16:22.874 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:22.876+0000 7f5fdbbeb7c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:16:22.874 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:22.876+0000 7f5fdbbeb7c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:16:22.918 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:22.920+0000 7f5fdbbeb7c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-04-25T07:16:33.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-04-25T07:16:33.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:16:33.068 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T07:16:33.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:16:33.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:16:33.617 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T07:16:33.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:16:33.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:16:33.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:16:33.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:16:33.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:16:33.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:33.652+0000 7fc3cdbd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:33.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:33.652+0000 7fc3cdbd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:33.651 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:33.652+0000 7fc3cdbd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:34.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:16:34.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:34.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:16:34.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:34.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:34.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:34.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:34.061 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:16:34.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:34.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:34.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:34.687 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:34.688+0000 7fc3cdbd97c0 -1 Falling back to public interface 2024-04-25T07:16:35.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:35.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:35.507 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:16:35.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:16:35.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:35.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:35.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:36.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:36.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:36.957 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:16:36.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:16:36.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:36.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:37.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:38.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:38.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:38.441 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:16:38.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:16:38.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:38.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:38.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:39.522 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:16:39.524+0000 7fc3cdbd97c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:16:39.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:39.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:39.913 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:16:39.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:16:39.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:39.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:40.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:41.421 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:16:41.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:16:41.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:41.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:41.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:42.858 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:16:42.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:16:42.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:42.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:43.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:44.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:44.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:44.396 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:16:44.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:16:44.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:44.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:44.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:45.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:45.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:45.827 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:16:45.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:16:45.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:45.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:47.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:16:47.422 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:16:47.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:47.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:47.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:48.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:48.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:48.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:16:48.911 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:16:48.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:48.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:49.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:50.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:50.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:50.369 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:16:50.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:16:50.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:50.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:50.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:51.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:51.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:51.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:16:51.788 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:16:51.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:51.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:52.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:53.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:53.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:53.664 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:16:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:16:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:54.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:55.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:55.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:55.176 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:16:55.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:16:55.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:55.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:55.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:56.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:56.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:56.967 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:16:56.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:16:56.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:56.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:57.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:58.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:58.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:58.752 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:16:58.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:16:58.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:58.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:16:59.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:00.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:00.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:00.522 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:17:00.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:17:00.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:00.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:17:01.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:02.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:02.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:02.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:17:02.279 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T07:17:02.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:02.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:17:02.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:03.698 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T07:17:03.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:17:03.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:03.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:17:04.126 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3345486069,v1:127.0.0.1:6827/3345486069] [v2:127.0.0.1:6828/3345486069,v1:127.0.0.1:6829/3345486069] exists,up d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5 2024-04-25T07:17:04.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:17:04.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:17:04.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:17:04.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:17:04.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=103631 2024-04-25T07:17:04.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-04-25T07:17:09.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-04-25T07:17:09.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:17:10.170 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T07:17:10.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:17:11.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:17:11.249 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T07:17:11.249 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:17:11.249 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_104990 2024-04-25T07:17:11.249 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:17:11.249 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:17:12.249 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 62 46 0.179676 0.179688 0.326978 0.321933 2024-04-25T07:17:13.249 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 105 89 0.173808 0.167969 0.262404 0.329999 2024-04-25T07:17:14.249 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 169 153 0.199194 0.25 0.265154 0.3098 2024-04-25T07:17:15.249 INFO:tasks.workunit.client.0.smithi088.stdout: 4 16 220 204 0.199193 0.199219 0.250217 0.298137 2024-04-25T07:17:16.249 INFO:tasks.workunit.client.0.smithi088.stdout: 5 16 284 268 0.209349 0.25 0.283698 0.288579 2024-04-25T07:17:17.249 INFO:tasks.workunit.client.0.smithi088.stdout: 6 16 345 329 0.214167 0.238281 0.284707 0.28775 2024-04-25T07:17:18.249 INFO:tasks.workunit.client.0.smithi088.stdout: 7 16 404 388 0.216493 0.230469 0.269672 0.284536 2024-04-25T07:17:19.250 INFO:tasks.workunit.client.0.smithi088.stdout: 8 16 471 455 0.222143 0.261719 0.254878 0.279524 2024-04-25T07:17:20.250 INFO:tasks.workunit.client.0.smithi088.stdout: 9 16 524 508 0.22046 0.207031 0.287369 0.277085 2024-04-25T07:17:21.250 INFO:tasks.workunit.client.0.smithi088.stdout: 10 16 598 582 0.227317 0.289062 0.237038 0.273451 2024-04-25T07:17:22.250 INFO:tasks.workunit.client.0.smithi088.stdout: 11 16 651 635 0.225471 0.207031 0.273947 0.273775 2024-04-25T07:17:23.250 INFO:tasks.workunit.client.0.smithi088.stdout: 12 16 711 695 0.22621 0.234375 0.274876 0.272881 2024-04-25T07:17:24.250 INFO:tasks.workunit.client.0.smithi088.stdout: 13 16 764 748 0.224732 0.207031 0.316096 0.273524 2024-04-25T07:17:25.251 INFO:tasks.workunit.client.0.smithi088.stdout: 14 16 823 807 0.22514 0.230469 0.268198 0.275102 2024-04-25T07:17:26.250 INFO:tasks.workunit.client.0.smithi088.stdout: 15 16 876 860 0.22393 0.207031 0.340388 0.275859 2024-04-25T07:17:27.251 INFO:tasks.workunit.client.0.smithi088.stdout: 16 16 940 924 0.225558 0.25 0.259158 0.274052 2024-04-25T07:17:28.251 INFO:tasks.workunit.client.0.smithi088.stdout: 17 16 1004 988 0.226994 0.25 0.255126 0.273997 2024-04-25T07:17:29.251 INFO:tasks.workunit.client.0.smithi088.stdout: 18 16 1063 1047 0.227185 0.230469 0.248738 0.273423 2024-04-25T07:17:30.251 INFO:tasks.workunit.client.0.smithi088.stdout: 19 16 1094 1078 0.221601 0.121094 0.287088 0.280209 2024-04-25T07:17:31.251 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:17:31.257061+0000 min lat: 0.159979 max lat: 0.744743 avg lat: 0.28076 2024-04-25T07:17:31.252 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:17:31.252 INFO:tasks.workunit.client.0.smithi088.stdout: 20 16 1145 1129 0.220481 0.199219 0.275993 0.28076 2024-04-25T07:17:32.251 INFO:tasks.workunit.client.0.smithi088.stdout: 21 16 1194 1178 0.219095 0.191406 0.276467 0.282826 2024-04-25T07:17:33.252 INFO:tasks.workunit.client.0.smithi088.stdout: 22 16 1258 1242 0.220498 0.25 0.24407 0.28186 2024-04-25T07:17:34.252 INFO:tasks.workunit.client.0.smithi088.stdout: 23 16 1322 1306 0.221779 0.25 0.276454 0.280605 2024-04-25T07:17:35.252 INFO:tasks.workunit.client.0.smithi088.stdout: 24 16 1386 1370 0.222954 0.25 0.250096 0.27957 2024-04-25T07:17:36.252 INFO:tasks.workunit.client.0.smithi088.stdout: 25 16 1442 1426 0.222785 0.21875 0.324808 0.2796 2024-04-25T07:17:37.252 INFO:tasks.workunit.client.0.smithi088.stdout: 26 16 1498 1482 0.222628 0.21875 0.252105 0.279152 2024-04-25T07:17:38.252 INFO:tasks.workunit.client.0.smithi088.stdout: 27 16 1555 1539 0.222628 0.222656 0.255665 0.278627 2024-04-25T07:17:39.252 INFO:tasks.workunit.client.0.smithi088.stdout: 28 16 1619 1603 0.223605 0.25 0.254852 0.277799 2024-04-25T07:17:40.252 INFO:tasks.workunit.client.0.smithi088.stdout: 29 16 1683 1667 0.224514 0.25 0.277909 0.277215 2024-04-25T07:17:41.252 INFO:tasks.workunit.client.0.smithi088.stdout: 30 16 1745 1729 0.225102 0.242188 0.273882 0.276478 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 30.219 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 1745 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4096 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4096 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.225567 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.033042 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.289062 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.121094 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 57 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 8.45876 2024-04-25T07:17:42.253 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 74 2024-04-25T07:17:42.254 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 31 2024-04-25T07:17:42.254 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.276398 2024-04-25T07:17:42.254 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0576124 2024-04-25T07:17:42.254 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.744743 2024-04-25T07:17:42.254 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.159979 2024-04-25T07:17:42.357 INFO:tasks.workunit.client.0.smithi088.stdout:after bench 2024-04-25T07:17:42.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-04-25T07:17:42.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 96078 2024-04-25T07:17:42.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-04-25T07:17:43.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 96078 2024-04-25T07:17:43.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (96078) - No such process 2024-04-25T07:17:43.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-04-25T07:17:43.796 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T07:17:43.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 98597 2024-04-25T07:17:43.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-04-25T07:17:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 98597 2024-04-25T07:17:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (98597) - No such process 2024-04-25T07:17:44.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-04-25T07:17:45.397 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T07:17:45.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 101115 2024-04-25T07:17:45.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-04-25T07:17:46.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 101115 2024-04-25T07:17:46.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (101115) - No such process 2024-04-25T07:17:46.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-04-25T07:17:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already down. 2024-04-25T07:17:46.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 103631 2024-04-25T07:17:46.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-04-25T07:17:47.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 103631 2024-04-25T07:17:48.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (103631) - No such process 2024-04-25T07:17:48.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-04-25T07:17:48.433 INFO:tasks.workunit.client.0.smithi088.stderr:osd.3 is already down. 2024-04-25T07:17:48.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:17:50.253 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:17:50.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:17:52.088 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:17:52.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:17:53.928 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:17:53.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:17:55.760 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:17:55.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-04-25T07:17:55.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-04-25T07:17:55.843 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:17:56.732 INFO:tasks.workunit.client.0.smithi088.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:17:56.733 INFO:tasks.workunit.client.0.smithi088.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:17:56.733 INFO:tasks.workunit.client.0.smithi088.stdout:2 : device size 0x100000000 : using 0x8054a000(2.0 GiB) 2024-04-25T07:17:56.733 INFO:tasks.workunit.client.0.smithi088.stdout:Expanding DB/WAL... 2024-04-25T07:17:56.733 INFO:tasks.workunit.client.0.smithi088.stdout:0 : expanding to 0x536870912 2024-04-25T07:17:56.756 INFO:tasks.workunit.client.0.smithi088.stdout:0 : size label updated to 536870912 2024-04-25T07:17:56.756 INFO:tasks.workunit.client.0.smithi088.stdout:1 : expanding to 0x1073741824 2024-04-25T07:17:56.781 INFO:tasks.workunit.client.0.smithi088.stdout:1 : size label updated to 1073741824 2024-04-25T07:17:56.781 INFO:tasks.workunit.client.0.smithi088.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-04-25T07:17:56.974 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:17:56.978+0000 7f2b0f8acac0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-04-25T07:17:56.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:17:56.978+0000 7f2b0f8acac0 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-04-25T07:18:02.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-04-25T07:18:02.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-04-25T07:18:02.381 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:03.282 INFO:tasks.workunit.client.0.smithi088.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:18:03.283 INFO:tasks.workunit.client.0.smithi088.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:18:03.283 INFO:tasks.workunit.client.0.smithi088.stdout:2 : device size 0x101000000 : using 0x81572000(2.0 GiB) 2024-04-25T07:18:03.283 INFO:tasks.workunit.client.0.smithi088.stdout:Expanding DB/WAL... 2024-04-25T07:18:03.283 INFO:tasks.workunit.client.0.smithi088.stdout:0 : expanding to 0x536870912 2024-04-25T07:18:03.313 INFO:tasks.workunit.client.0.smithi088.stdout:0 : size label updated to 536870912 2024-04-25T07:18:03.313 INFO:tasks.workunit.client.0.smithi088.stdout:1 : expanding to 0x1073741824 2024-04-25T07:18:03.338 INFO:tasks.workunit.client.0.smithi088.stdout:1 : size label updated to 1073741824 2024-04-25T07:18:03.338 INFO:tasks.workunit.client.0.smithi088.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-04-25T07:18:03.581 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:03.586+0000 7f330d032ac0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-04-25T07:18:03.581 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:03.586+0000 7f330d032ac0 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-04-25T07:18:08.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-04-25T07:18:08.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-04-25T07:18:08.686 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:09.591 INFO:tasks.workunit.client.0.smithi088.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:18:09.591 INFO:tasks.workunit.client.0.smithi088.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:18:09.591 INFO:tasks.workunit.client.0.smithi088.stdout:2 : device size 0x100020000 : using 0x80580000(2.0 GiB) 2024-04-25T07:18:09.591 INFO:tasks.workunit.client.0.smithi088.stdout:Expanding DB/WAL... 2024-04-25T07:18:09.591 INFO:tasks.workunit.client.0.smithi088.stdout:0 : expanding to 0x536870912 2024-04-25T07:18:09.619 INFO:tasks.workunit.client.0.smithi088.stdout:0 : size label updated to 536870912 2024-04-25T07:18:09.619 INFO:tasks.workunit.client.0.smithi088.stdout:1 : expanding to 0x1073741824 2024-04-25T07:18:09.644 INFO:tasks.workunit.client.0.smithi088.stdout:1 : size label updated to 1073741824 2024-04-25T07:18:09.644 INFO:tasks.workunit.client.0.smithi088.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-04-25T07:18:09.845 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:09.850+0000 7fc5ba310ac0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-04-25T07:18:09.845 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:09.850+0000 7fc5ba310ac0 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-04-25T07:18:15.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-04-25T07:18:15.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-04-25T07:18:15.089 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:15.995 INFO:tasks.workunit.client.0.smithi088.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:18:15.995 INFO:tasks.workunit.client.0.smithi088.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:18:15.995 INFO:tasks.workunit.client.0.smithi088.stdout:2 : device size 0xfff00000 : using 0x8045a000(2.0 GiB) 2024-04-25T07:18:15.995 INFO:tasks.workunit.client.0.smithi088.stdout:Expanding DB/WAL... 2024-04-25T07:18:15.995 INFO:tasks.workunit.client.0.smithi088.stdout:0 : expanding to 0x536870912 2024-04-25T07:18:16.034 INFO:tasks.workunit.client.0.smithi088.stdout:0 : size label updated to 536870912 2024-04-25T07:18:16.034 INFO:tasks.workunit.client.0.smithi088.stdout:1 : expanding to 0x1073741824 2024-04-25T07:18:16.076 INFO:tasks.workunit.client.0.smithi088.stdout:1 : size label updated to 1073741824 2024-04-25T07:18:16.076 INFO:tasks.workunit.client.0.smithi088.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-04-25T07:18:16.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:16.265+0000 7f09a4b66ac0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-04-25T07:18:16.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:16.265+0000 7f09a4b66ac0 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-04-25T07:18:21.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:18:23.405 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:23.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:18:25.233 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:25.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:18:27.064 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:27.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:18:28.896 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:28.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-04-25T07:18:28.944 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:30.726 INFO:tasks.workunit.client.0.smithi088.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-04-25T07:18:30.727 INFO:tasks.workunit.client.0.smithi088.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-04-25T07:18:30.727 INFO:tasks.workunit.client.0.smithi088.stdout:2 : device size 0x100000000 : using 0x54a000(5.3 MiB) 2024-04-25T07:18:30.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-04-25T07:18:30.755 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:32.536 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-04-25T07:18:32.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:18:33.856 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:33.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-04-25T07:18:33.903 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:35.685 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:18:35.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:18:37.000 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-04-25T07:18:37.048 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:38.829 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-04-25T07:18:38.829 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-04-25T07:18:38.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:18:39.649 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:39.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-04-25T07:18:39.689 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:39.689 INFO:tasks.workunit.client.0.smithi088.stderr:Migrate to WAL device isn't supported. 2024-04-25T07:18:39.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-04-25T07:18:39.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:18:41.500 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:41.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-04-25T07:18:41.550 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:43.588 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-04-25T07:18:43.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:18:44.904 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:44.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-04-25T07:18:44.950 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:18:46.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:18:47.548 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:18:47.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:18:47.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:18:47.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:18:47.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:18:47.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:18:47.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:18:47.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:18:47.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:18:47.574 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:18:47.574 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:47.574 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:18:47.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:18:47.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:18:47.576 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:18:47.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:18:47.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:18:47.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:18:47.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:18:47.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:18:47.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:18:47.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:18:47.615 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:47.617+0000 7f895e4747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:47.618 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:47.621+0000 7f895e4747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:47.620 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:47.621+0000 7f895e4747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:48.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:18:48.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:48.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:18:48.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:48.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:48.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:48.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:48.047 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:18:48.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:18:48.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:48.403 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:48.405+0000 7f895e4747c0 -1 Falling back to public interface 2024-04-25T07:18:48.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:49.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:49.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:49.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:18:49.483 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:18:49.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:49.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:18:49.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:50.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:50.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:50.938 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:18:50.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:18:50.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:50.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:18:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:52.324 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:52.325+0000 7f895e4747c0 -1 osd.0 26 log_to_monitors true 2024-04-25T07:18:52.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:52.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:52.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:18:52.348 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:18:52.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:18:52.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:52.799 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 37 up_thru 21 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/2456472203,v1:127.0.0.1:6803/2456472203] [v2:127.0.0.1:6804/2456472203,v1:127.0.0.1:6805/2456472203] exists,up 85e4cd7e-efc8-49c2-a5ee-9fe41f509b63 2024-04-25T07:18:52.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:52.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:52.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:52.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=117066 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:18:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:18:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:18:52.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:18:52.807 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:18:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:18:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:18:52.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:18:52.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:18:52.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:18:52.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:18:52.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:18:52.831 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:52.833+0000 7fd4441e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:52.864 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:52.869+0000 7fd4441e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:52.866 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:52.869+0000 7fd4441e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:18:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:18:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:53.284 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:18:53.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:53.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:53.673+0000 7fd4441e27c0 -1 Falling back to public interface 2024-04-25T07:18:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:54.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:54.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:54.701 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:18:54.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:18:54.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:54.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:55.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:56.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:56.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:56.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:18:56.166 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:18:56.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:56.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:56.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:57.492 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:57.493+0000 7fd4441e27c0 -1 osd.1 28 log_to_monitors true 2024-04-25T07:18:57.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:57.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:57.585 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:18:57.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:18:57.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:57.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:58.185 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 40 up_thru 27 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6810/3553500491,v1:127.0.0.1:6811/3553500491] [v2:127.0.0.1:6812/3553500491,v1:127.0.0.1:6813/3553500491] exists,up 1dd411e3-2838-4aa5-96cc-e48db2d17c8a 2024-04-25T07:18:58.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:58.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:58.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:58.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=118005 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:18:58.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:18:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:18:58.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:18:58.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:18:58.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:18:58.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:18:58.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:18:58.194 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:18:58.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:18:58.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:18:58.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:18:58.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:18:58.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:18:58.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:18:58.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:18:58.223 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:58.225+0000 7f68d4f797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:58.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:58.345+0000 7f68d4f797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:58.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:58.349+0000 7f68d4f797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:58.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:18:58.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:58.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:18:58.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:58.732 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:18:58.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:58.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:58.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:58.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:58.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:58.894 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:18:58.897+0000 7f68d4f797c0 -1 Falling back to public interface 2024-04-25T07:18:59.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:00.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:00.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:00.243 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:19:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:19:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:00.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:01.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:01.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:01.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:19:01.655 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:19:01.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:01.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:01.879 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:19:01.881+0000 7f68d4f797c0 -1 osd.2 32 log_to_monitors true 2024-04-25T07:19:02.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:03.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:03.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:03.088 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:19:03.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:19:03.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:03.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:03.766 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 44 up_thru 44 down_at 33 last_clean_interval [14,32) [v2:127.0.0.1:6818/4162021192,v1:127.0.0.1:6819/4162021192] [v2:127.0.0.1:6820/4162021192,v1:127.0.0.1:6821/4162021192] exists,up ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6 2024-04-25T07:19:03.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:19:03.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:19:03.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:19:03.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=118959 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:19:03.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:03.771 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:19:03.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:19:03.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:19:03.774 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:19:03.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:19:03.797 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:19:03.800+0000 7f57750547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:03.799 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:19:03.800+0000 7f57750547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:03.800 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:19:03.804+0000 7f57750547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:04.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:19:04.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:19:04.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:19:04.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:19:04.224 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:19:04.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:19:04.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:04.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:19:04.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:04.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:19:04.582 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:19:04.584+0000 7f57750547c0 -1 Falling back to public interface 2024-04-25T07:19:04.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:05.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:05.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:05.726 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:19:05.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:19:05.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:05.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:19:06.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:07.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:07.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:07.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:19:07.186 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:19:07.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:07.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:19:07.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:08.554 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:19:08.556+0000 7f57750547c0 -1 osd.3 34 log_to_monitors true 2024-04-25T07:19:08.582 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:19:08.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:08.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:08.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:19:08.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:08.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:19:09.040 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 48 up_thru 33 down_at 35 last_clean_interval [19,34) [v2:127.0.0.1:6826/2021386839,v1:127.0.0.1:6827/2021386839] [v2:127.0.0.1:6828/2021386839,v1:127.0.0.1:6829/2021386839] exists,up d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5 2024-04-25T07:19:09.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:19:09.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:19:09.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:19:09.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:19:09.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=119878 2024-04-25T07:19:09.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-04-25T07:19:09.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:19:09.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:19:09.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:19:09.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:19:09.043 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:19:09.043 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:19:09.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:19:09.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:19:09.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:19:09.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:19:09.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:19:09.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:19:09.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:19:09.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:19:09.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:19:09.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:19:09.849 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:19:09.849 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:19:09.850 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T07:19:09.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:19:09.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:09.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:19:10.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789958 2024-04-25T07:19:10.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789958 2024-04-25T07:19:10.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789958' 2024-04-25T07:19:10.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:10.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:19:10.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=171798691845 2024-04-25T07:19:10.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 171798691845 2024-04-25T07:19:10.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789958 1-171798691845' 2024-04-25T07:19:10.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:10.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:19:10.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561028 2024-04-25T07:19:10.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561028 2024-04-25T07:19:10.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789958 1-171798691845 2-188978561028' 2024-04-25T07:19:10.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:10.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:19:10.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=206158430211 2024-04-25T07:19:10.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 206158430211 2024-04-25T07:19:10.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789958 1-171798691845 2-188978561028 3-206158430211' 2024-04-25T07:19:10.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:10.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:10.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-158913789958 2024-04-25T07:19:10.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:19:10.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-158913789958 2024-04-25T07:19:10.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:10.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789958 2024-04-25T07:19:10.620 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 158913789958 2024-04-25T07:19:10.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 158913789958' 2024-04-25T07:19:10.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:19:11.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789956 -lt 158913789958 2024-04-25T07:19:11.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:19:12.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:19:12.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:19:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789958 -lt 158913789958 2024-04-25T07:19:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:12.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-171798691845 2024-04-25T07:19:12.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:12.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:19:12.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-171798691845 2024-04-25T07:19:12.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:12.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=171798691845 2024-04-25T07:19:12.464 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 171798691845 2024-04-25T07:19:12.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-04-25T07:19:12.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:19:12.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 171798691845 -lt 171798691845 2024-04-25T07:19:12.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:12.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-188978561028 2024-04-25T07:19:12.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:12.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:19:12.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-188978561028 2024-04-25T07:19:12.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:12.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561028 2024-04-25T07:19:12.931 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 188978561028 2024-04-25T07:19:12.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 188978561028' 2024-04-25T07:19:12.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:19:13.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561028 -lt 188978561028 2024-04-25T07:19:13.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:13.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-206158430211 2024-04-25T07:19:13.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:13.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:19:13.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-206158430211 2024-04-25T07:19:13.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:13.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=206158430211 2024-04-25T07:19:13.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 206158430211' 2024-04-25T07:19:13.387 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 206158430211 2024-04-25T07:19:13.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:19:13.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 206158430211 -lt 206158430211 2024-04-25T07:19:13.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:19:13.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:13.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:14.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:19:14.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:14.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:14.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:14.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:14.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:14.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:14.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:14.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:19:14.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:14.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:14.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:15.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:19:15.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:19:15.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:19:15.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:19:15.418 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T07:19:15.418 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:19:15.418 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_121185 2024-04-25T07:19:15.419 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:19:15.419 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:19:16.418 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 69 53 0.207018 0.207031 0.22392 0.272711 2024-04-25T07:19:17.418 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 119 103 0.201155 0.195312 0.147523 0.282086 2024-04-25T07:19:18.418 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 177 161 0.209617 0.226562 0.183581 0.281955 2024-04-25T07:19:19.419 INFO:tasks.workunit.client.0.smithi088.stdout: 4 16 235 219 0.213844 0.226562 0.114346 0.279374 2024-04-25T07:19:20.419 INFO:tasks.workunit.client.0.smithi088.stdout: 5 16 289 273 0.213257 0.210938 0.286637 0.282684 2024-04-25T07:19:21.419 INFO:tasks.workunit.client.0.smithi088.stdout: 6 16 349 333 0.216772 0.234375 0.297562 0.282542 2024-04-25T07:19:22.419 INFO:tasks.workunit.client.0.smithi088.stdout: 7 16 403 387 0.215934 0.210938 0.243758 0.28447 2024-04-25T07:19:23.419 INFO:tasks.workunit.client.0.smithi088.stdout: 8 16 461 445 0.21726 0.226562 0.18605 0.282535 2024-04-25T07:19:24.419 INFO:tasks.workunit.client.0.smithi088.stdout: 9 16 523 507 0.220026 0.242188 0.187115 0.279161 2024-04-25T07:19:25.420 INFO:tasks.workunit.client.0.smithi088.stdout: 10 16 581 565 0.220676 0.226562 0.263821 0.279374 2024-04-25T07:19:26.420 INFO:tasks.workunit.client.0.smithi088.stdout: 11 16 643 627 0.222629 0.242188 0.301631 0.277797 2024-04-25T07:19:27.420 INFO:tasks.workunit.client.0.smithi088.stdout: 12 16 704 688 0.22393 0.238281 0.191287 0.275648 2024-04-25T07:19:28.420 INFO:tasks.workunit.client.0.smithi088.stdout: 13 16 747 731 0.219624 0.167969 0.266547 0.280568 2024-04-25T07:19:29.420 INFO:tasks.workunit.client.0.smithi088.stdout: 14 16 809 793 0.221234 0.242188 0.213814 0.280042 2024-04-25T07:19:30.420 INFO:tasks.workunit.client.0.smithi088.stdout: 15 16 864 848 0.220806 0.214844 0.192515 0.28039 2024-04-25T07:19:31.420 INFO:tasks.workunit.client.0.smithi088.stdout: 16 16 924 908 0.221653 0.234375 0.285272 0.280565 2024-04-25T07:19:32.421 INFO:tasks.workunit.client.0.smithi088.stdout: 17 16 993 977 0.224467 0.269531 0.185076 0.276376 2024-04-25T07:19:33.421 INFO:tasks.workunit.client.0.smithi088.stdout: 18 16 1058 1042 0.2261 0.253906 0.20046 0.272139 2024-04-25T07:19:34.421 INFO:tasks.workunit.client.0.smithi088.stdout: 19 16 1132 1116 0.229413 0.289062 0.284145 0.270802 2024-04-25T07:19:35.421 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:19:35.426398+0000 min lat: 0.113836 max lat: 0.552429 avg lat: 0.266407 2024-04-25T07:19:35.421 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:19:35.421 INFO:tasks.workunit.client.0.smithi088.stdout: 20 16 1213 1197 0.23376 0.316406 0.226466 0.266407 2024-04-25T07:19:36.421 INFO:tasks.workunit.client.0.smithi088.stdout: 21 16 1287 1271 0.236392 0.289062 0.256118 0.263351 2024-04-25T07:19:37.421 INFO:tasks.workunit.client.0.smithi088.stdout: 22 16 1361 1345 0.238785 0.289062 0.218091 0.260266 2024-04-25T07:19:38.421 INFO:tasks.workunit.client.0.smithi088.stdout: 23 16 1441 1425 0.241988 0.3125 0.194641 0.257022 2024-04-25T07:19:39.422 INFO:tasks.workunit.client.0.smithi088.stdout: 24 16 1505 1489 0.24232 0.25 0.226388 0.256961 2024-04-25T07:19:40.422 INFO:tasks.workunit.client.0.smithi088.stdout: 25 16 1572 1556 0.243095 0.261719 0.213913 0.256139 2024-04-25T07:19:41.422 INFO:tasks.workunit.client.0.smithi088.stdout: 26 16 1642 1626 0.244258 0.273438 0.169879 0.254983 2024-04-25T07:19:42.422 INFO:tasks.workunit.client.0.smithi088.stdout: 27 16 1714 1698 0.245626 0.28125 0.198888 0.25373 2024-04-25T07:19:43.422 INFO:tasks.workunit.client.0.smithi088.stdout: 28 16 1796 1780 0.248292 0.320312 0.21244 0.250797 2024-04-25T07:19:44.422 INFO:tasks.workunit.client.0.smithi088.stdout: 29 16 1863 1847 0.248754 0.261719 0.180168 0.250194 2024-04-25T07:19:45.423 INFO:tasks.workunit.client.0.smithi088.stdout: 30 16 1940 1924 0.250487 0.300781 0.230255 0.248555 2024-04-25T07:19:46.423 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 30.1626 2024-04-25T07:19:46.423 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 1940 2024-04-25T07:19:46.423 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4096 2024-04-25T07:19:46.423 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4096 2024-04-25T07:19:46.423 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.251242 2024-04-25T07:19:46.423 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.0378132 2024-04-25T07:19:46.423 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.320312 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.167969 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 64 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 9.68017 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 82 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 43 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.248275 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0637202 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.552429 2024-04-25T07:19:46.424 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.113836 2024-04-25T07:19:46.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 117066 2024-04-25T07:19:46.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-04-25T07:19:47.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 117066 2024-04-25T07:19:47.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (117066) - No such process 2024-04-25T07:19:47.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-04-25T07:19:48.008 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T07:19:48.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 118005 2024-04-25T07:19:48.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-04-25T07:19:49.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 118005 2024-04-25T07:19:49.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (118005) - No such process 2024-04-25T07:19:49.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-04-25T07:19:49.511 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T07:19:49.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 118959 2024-04-25T07:19:49.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-04-25T07:19:50.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 118959 2024-04-25T07:19:50.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (118959) - No such process 2024-04-25T07:19:50.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-04-25T07:19:50.984 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already down. 2024-04-25T07:19:51.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 119878 2024-04-25T07:19:51.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-04-25T07:19:52.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 119878 2024-04-25T07:19:52.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (119878) - No such process 2024-04-25T07:19:52.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-04-25T07:19:52.519 INFO:tasks.workunit.client.0.smithi088.stderr:osd.3 is already down. 2024-04-25T07:19:52.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:19:53.841 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:19:53.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-04-25T07:19:54.167 INFO:tasks.workunit.client.0.smithi088.stderr:512+0 records in 2024-04-25T07:19:54.167 INFO:tasks.workunit.client.0.smithi088.stderr:512+0 records out 2024-04-25T07:19:54.167 INFO:tasks.workunit.client.0.smithi088.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.303476 s, 1.8 GB/s 2024-04-25T07:19:54.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-04-25T07:19:54.193 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:19:58.725 INFO:tasks.workunit.client.0.smithi088.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-04-25T07:19:58.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:20:00.549 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:20:00.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:20:01.877 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:20:01.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-04-25T07:20:02.455 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records in 2024-04-25T07:20:02.455 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records out 2024-04-25T07:20:02.455 INFO:tasks.workunit.client.0.smithi088.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.561728 s, 1.9 GB/s 2024-04-25T07:20:02.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-04-25T07:20:02.480 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:20:10.265 INFO:tasks.workunit.client.0.smithi088.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-04-25T07:20:10.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-04-25T07:20:10.297 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:20:12.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:20:13.917 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:20:13.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:20:14.749 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:20:14.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-04-25T07:20:14.784 INFO:tasks.workunit.client.0.smithi088.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:20:14.784 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:20:15.565 INFO:tasks.workunit.client.0.smithi088.stdout:DB device added 2024-04-25T07:20:15.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-04-25T07:20:15.597 INFO:tasks.workunit.client.0.smithi088.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:20:15.597 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:20:16.889 INFO:tasks.workunit.client.0.smithi088.stdout:WAL device added 2024-04-25T07:20:16.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-04-25T07:20:16.920 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:20:19.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:20:21.029 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:20:21.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:20:22.365 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:20:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-04-25T07:20:22.407 INFO:tasks.workunit.client.0.smithi088.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:20:22.407 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:20:23.689 INFO:tasks.workunit.client.0.smithi088.stdout:WAL device added 2024-04-25T07:20:23.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-04-25T07:20:23.724 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:20:25.757 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-04-25T07:20:25.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:20:27.077 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:20:27.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:20:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:20:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:20:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:20:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:20:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:20:27.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:20:27.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:20:27.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:20:27.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:20:27.098 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:20:27.098 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:20:27.098 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:20:27.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:20:27.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:20:27.101 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:20:27.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:20:27.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:20:27.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:20:27.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:20:27.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:20:27.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:20:27.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:20:27.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:27.138+0000 7fbfe322c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:27.142 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:27.142+0000 7fbfe322c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:27.144 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:27.142+0000 7fbfe322c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:27.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:20:27.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:20:27.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:20:27.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:20:27.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:20:27.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:27.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:20:27.521 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:20:27.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:27.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:27.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:28.188 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:28.186+0000 7fbfe322c7c0 -1 Falling back to public interface 2024-04-25T07:20:28.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:28.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:28.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:20:28.910 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:20:28.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:28.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:29.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:30.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:30.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:30.324 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:20:30.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:20:30.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:30.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:30.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:31.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:31.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:31.784 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:20:31.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:20:31.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:31.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:32.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:32.999 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:32.998+0000 7fbfe322c7c0 -1 osd.0 50 log_to_monitors true 2024-04-25T07:20:33.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:33.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:33.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:20:33.230 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:20:33.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:33.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:33.708 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 61 up_thru 61 down_at 51 last_clean_interval [37,50) [v2:127.0.0.1:6802/2208213255,v1:127.0.0.1:6803/2208213255] [v2:127.0.0.1:6804/2208213255,v1:127.0.0.1:6805/2208213255] exists,up 85e4cd7e-efc8-49c2-a5ee-9fe41f509b63 2024-04-25T07:20:33.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:20:33.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:20:33.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:20:33.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:20:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=127294 2024-04-25T07:20:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:20:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:20:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:20:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:20:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:20:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:20:33.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:20:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:20:33.715 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:20:33.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:20:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:20:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:20:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:20:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:20:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:20:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:20:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:20:33.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:20:33.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:20:33.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:20:33.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:20:33.717 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:20:33.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:20:33.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:20:33.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:20:33.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:20:33.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:20:33.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:20:33.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:20:33.741 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:33.738+0000 7ff7be8a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:33.743 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:33.742+0000 7ff7be8a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:33.745 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:33.742+0000 7ff7be8a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:34.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:20:34.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:20:34.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:20:34.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:20:34.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:20:34.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:34.228 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:20:34.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:20:34.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:34.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:34.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:34.804 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:34.802+0000 7ff7be8a77c0 -1 Falling back to public interface 2024-04-25T07:20:35.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:35.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:35.703 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:20:35.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:20:35.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:35.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:36.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:37.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:37.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:37.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:20:37.100 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:20:37.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:37.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:37.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:38.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:38.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:38.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:20:38.526 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:20:38.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:38.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:38.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:39.587 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:39.590+0000 7ff7be8a77c0 -1 osd.1 53 log_to_monitors true 2024-04-25T07:20:39.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:40.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:40.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:20:40.000 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:20:40.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:40.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:20:40.484 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 64 up_thru 64 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/1997991544,v1:127.0.0.1:6811/1997991544] [v2:127.0.0.1:6812/1997991544,v1:127.0.0.1:6813/1997991544] exists,up 1dd411e3-2838-4aa5-96cc-e48db2d17c8a 2024-04-25T07:20:40.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:20:40.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:20:40.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:20:40.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:20:40.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=128263 2024-04-25T07:20:40.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:20:40.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:20:40.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:20:40.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:20:40.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:20:40.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:20:40.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:20:40.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:20:40.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:20:40.491 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:20:40.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:20:40.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:20:40.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:20:40.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:20:40.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:20:40.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:20:40.519 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:40.522+0000 7f4486ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:40.624 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:40.626+0000 7f4486ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:40.625 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:40.630+0000 7f4486ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:20:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:20:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:20:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:20:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:20:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:20:41.048 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:20:41.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:41.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:41.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:41.671 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:41.674+0000 7f4486ee47c0 -1 Falling back to public interface 2024-04-25T07:20:42.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:42.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:42.470 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:20:42.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:20:42.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:42.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:42.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:43.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:43.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:43.887 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:20:43.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:20:43.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:43.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:44.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:45.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:45.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:45.313 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:20:45.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:20:45.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:45.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:45.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:46.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:46.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:46.787 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:20:46.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:20:46.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:46.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:47.092 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:47.093+0000 7f4486ee47c0 -1 osd.2 56 log_to_monitors true 2024-04-25T07:20:47.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:48.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:48.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:48.202 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:20:48.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:20:48.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:48.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:48.948 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 68 up_thru 54 down_at 57 last_clean_interval [44,56) [v2:127.0.0.1:6818/2717538032,v1:127.0.0.1:6819/2717538032] [v2:127.0.0.1:6820/2717538032,v1:127.0.0.1:6821/2717538032] exists,up ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6 2024-04-25T07:20:48.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:20:48.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:20:48.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:20:48.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:20:48.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=129249 2024-04-25T07:20:48.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:20:48.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:20:48.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:20:48.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:20:48.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:20:48.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:20:48.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:20:48.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:20:48.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:20:48.956 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:20:48.956 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:20:48.956 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:20:48.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:20:48.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:20:48.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:20:48.957 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T07:20:48.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:20:48.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:20:48.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:20:48.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:20:48.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:20:48.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:20:48.982 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:48.985+0000 7fbe5321c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:48.984 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:48.985+0000 7fbe5321c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:48.986 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:48.989+0000 7fbe5321c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:49.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:20:49.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:20:49.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:20:49.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:20:49.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:20:49.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:49.641 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:20:49.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:20:49.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:49.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:20:49.822 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:49.825+0000 7fbe5321c7c0 -1 Falling back to public interface 2024-04-25T07:20:50.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:51.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:51.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:51.079 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:20:51.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:20:51.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:51.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:20:51.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:52.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:52.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:52.504 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:20:52.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:20:52.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:52.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:20:52.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:53.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:53.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:53.921 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:20:53.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:20:53.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:53.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:20:54.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:54.555 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:20:54.557+0000 7fbe5321c7c0 -1 osd.3 58 log_to_monitors true 2024-04-25T07:20:55.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:55.330 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:20:55.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:55.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:20:55.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:55.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:20:56.129 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 73 up_thru 57 down_at 59 last_clean_interval [48,58) [v2:127.0.0.1:6826/4031628383,v1:127.0.0.1:6827/4031628383] [v2:127.0.0.1:6828/4031628383,v1:127.0.0.1:6829/4031628383] exists,up d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5 2024-04-25T07:20:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:20:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:20:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:20:56.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:20:56.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=130227 2024-04-25T07:20:56.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:20:56.323 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T07:20:56.323 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:20:56.323 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_131176 2024-04-25T07:20:56.323 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:20:56.323 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:20:57.323 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 40 24 0.0937429 0.09375 0.28284 0.491797 2024-04-25T07:20:58.323 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 93 77 0.150374 0.207031 0.289217 0.378301 2024-04-25T07:20:59.323 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 140 124 0.161441 0.183594 0.329106 0.359364 2024-04-25T07:21:00.323 INFO:tasks.workunit.client.0.smithi088.stdout: 4 16 190 174 0.169904 0.195312 0.254492 0.353763 2024-04-25T07:21:01.323 INFO:tasks.workunit.client.0.smithi088.stdout: 5 16 234 218 0.170292 0.171875 0.350562 0.350493 2024-04-25T07:21:02.323 INFO:tasks.workunit.client.0.smithi088.stdout: 6 16 282 266 0.173157 0.1875 0.351881 0.351473 2024-04-25T07:21:03.323 INFO:tasks.workunit.client.0.smithi088.stdout: 7 16 329 313 0.174643 0.183594 0.368225 0.350262 2024-04-25T07:21:04.324 INFO:tasks.workunit.client.0.smithi088.stdout: 8 16 380 364 0.177712 0.199219 0.207221 0.344658 2024-04-25T07:21:05.324 INFO:tasks.workunit.client.0.smithi088.stdout: 9 16 432 416 0.180532 0.203125 0.212365 0.340258 2024-04-25T07:21:06.324 INFO:tasks.workunit.client.0.smithi088.stdout: 10 16 476 460 0.179664 0.171875 0.421769 0.339231 2024-04-25T07:21:07.324 INFO:tasks.workunit.client.0.smithi088.stdout: 11 16 518 502 0.178243 0.164062 0.292123 0.347843 2024-04-25T07:21:08.324 INFO:tasks.workunit.client.0.smithi088.stdout: 12 16 563 547 0.178036 0.175781 0.236693 0.345547 2024-04-25T07:21:09.324 INFO:tasks.workunit.client.0.smithi088.stdout: 13 16 613 597 0.179363 0.195312 0.319506 0.344163 2024-04-25T07:21:10.325 INFO:tasks.workunit.client.0.smithi088.stdout: 14 16 656 640 0.178547 0.167969 0.36829 0.343955 2024-04-25T07:21:11.325 INFO:tasks.workunit.client.0.smithi088.stdout: 15 16 706 690 0.179663 0.195312 0.342586 0.343919 2024-04-25T07:21:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: 16 16 748 732 0.178686 0.164062 0.415466 0.345673 2024-04-25T07:21:13.325 INFO:tasks.workunit.client.0.smithi088.stdout: 17 16 799 783 0.179892 0.199219 0.339368 0.344518 2024-04-25T07:21:14.325 INFO:tasks.workunit.client.0.smithi088.stdout: 18 16 843 827 0.179445 0.171875 0.270874 0.344725 2024-04-25T07:21:15.325 INFO:tasks.workunit.client.0.smithi088.stdout: 19 16 895 879 0.180691 0.203125 0.316341 0.343522 2024-04-25T07:21:16.326 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:21:16.330368+0000 min lat: 0.18456 max lat: 0.692949 avg lat: 0.342919 2024-04-25T07:21:16.326 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:21:16.326 INFO:tasks.workunit.client.0.smithi088.stdout: 20 16 937 921 0.179858 0.164062 0.389558 0.342919 2024-04-25T07:21:17.326 INFO:tasks.workunit.client.0.smithi088.stdout: 21 16 988 972 0.180779 0.199219 0.236254 0.343128 2024-04-25T07:21:18.326 INFO:tasks.workunit.client.0.smithi088.stdout: 22 16 1036 1020 0.181083 0.1875 0.357568 0.342683 2024-04-25T07:21:19.326 INFO:tasks.workunit.client.0.smithi088.stdout: 23 16 1090 1074 0.18238 0.210938 0.323383 0.340737 2024-04-25T07:21:20.326 INFO:tasks.workunit.client.0.smithi088.stdout: 24 16 1137 1121 0.18243 0.183594 0.318849 0.340425 2024-04-25T07:21:21.326 INFO:tasks.workunit.client.0.smithi088.stdout: 25 16 1185 1169 0.182632 0.1875 0.245696 0.339616 2024-04-25T07:21:22.326 INFO:tasks.workunit.client.0.smithi088.stdout: 26 16 1232 1216 0.182668 0.183594 0.467171 0.339857 2024-04-25T07:21:23.326 INFO:tasks.workunit.client.0.smithi088.stdout: 27 16 1283 1267 0.18328 0.199219 0.297935 0.339048 2024-04-25T07:21:24.326 INFO:tasks.workunit.client.0.smithi088.stdout: 28 16 1334 1318 0.183848 0.199219 0.305236 0.33848 2024-04-25T07:21:25.330 INFO:tasks.workunit.client.0.smithi088.stdout: 29 16 1382 1366 0.183973 0.1875 0.253351 0.338067 2024-04-25T07:21:26.327 INFO:tasks.workunit.client.0.smithi088.stdout: 30 16 1434 1418 0.184611 0.203125 0.311529 0.337809 2024-04-25T07:21:27.327 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 30.289 2024-04-25T07:21:27.327 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 1434 2024-04-25T07:21:27.327 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4096 2024-04-25T07:21:27.327 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4096 2024-04-25T07:21:27.327 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.184937 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.022 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.210938 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.09375 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 47 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 5.63201 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 54 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 24 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.337421 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0689033 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.692949 2024-04-25T07:21:27.328 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.182573 2024-04-25T07:21:27.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 127294 2024-04-25T07:21:27.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-04-25T07:21:28.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 127294 2024-04-25T07:21:28.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (127294) - No such process 2024-04-25T07:21:28.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-04-25T07:21:28.986 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T07:21:29.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 128263 2024-04-25T07:21:29.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-04-25T07:21:30.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 128263 2024-04-25T07:21:30.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (128263) - No such process 2024-04-25T07:21:30.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-04-25T07:21:30.589 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T07:21:30.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 129249 2024-04-25T07:21:30.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-04-25T07:21:31.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 129249 2024-04-25T07:21:31.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (129249) - No such process 2024-04-25T07:21:31.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-04-25T07:21:32.148 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already down. 2024-04-25T07:21:32.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 130227 2024-04-25T07:21:32.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-04-25T07:21:33.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 130227 2024-04-25T07:21:33.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (130227) - No such process 2024-04-25T07:21:33.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-04-25T07:21:33.737 INFO:tasks.workunit.client.0.smithi088.stderr:osd.3 is already down. 2024-04-25T07:21:33.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:21:35.558 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:21:35.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-04-25T07:21:36.147 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records in 2024-04-25T07:21:36.147 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records out 2024-04-25T07:21:36.147 INFO:tasks.workunit.client.0.smithi088.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.563956 s, 1.9 GB/s 2024-04-25T07:21:36.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-04-25T07:21:36.171 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:21:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-04-25T07:21:45.458 INFO:tasks.workunit.client.0.smithi088.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-04-25T07:21:45.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:21:47.274 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:21:47.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-04-25T07:21:47.597 INFO:tasks.workunit.client.0.smithi088.stderr:512+0 records in 2024-04-25T07:21:47.597 INFO:tasks.workunit.client.0.smithi088.stderr:512+0 records out 2024-04-25T07:21:47.597 INFO:tasks.workunit.client.0.smithi088.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.302773 s, 1.8 GB/s 2024-04-25T07:21:47.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-04-25T07:21:47.616 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:21:52.906 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-04-25T07:21:52.906 INFO:tasks.workunit.client.0.smithi088.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-04-25T07:21:52.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-04-25T07:21:52.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:21:54.790 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:21:54.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:21:56.618 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:21:56.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-04-25T07:21:57.198 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records in 2024-04-25T07:21:57.198 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records out 2024-04-25T07:21:57.198 INFO:tasks.workunit.client.0.smithi088.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.564224 s, 1.9 GB/s 2024-04-25T07:21:57.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-04-25T07:21:57.222 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:22:06.266 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:22:06.266 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-04-25T07:22:06.266 INFO:tasks.workunit.client.0.smithi088.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-04-25T07:22:06.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-04-25T07:22:06.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:22:07.698 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:22:07.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-04-25T07:22:07.741 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:22:09.022 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:22:09.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-04-25T07:22:09.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:22:09.954 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:22:09.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-04-25T07:22:11.782 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:22:11.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-04-25T07:22:12.380 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records in 2024-04-25T07:22:12.380 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records out 2024-04-25T07:22:12.380 INFO:tasks.workunit.client.0.smithi088.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.572069 s, 1.9 GB/s 2024-04-25T07:22:12.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-04-25T07:22:12.403 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:22:12.403 INFO:tasks.workunit.client.0.smithi088.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-04-25T07:22:12.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-04-25T07:22:12.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:22:14.206 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:22:14.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-04-25T07:22:21.635 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records in 2024-04-25T07:22:21.635 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records out 2024-04-25T07:22:21.635 INFO:tasks.workunit.client.0.smithi088.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.27006 s, 148 MB/s 2024-04-25T07:22:21.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-04-25T07:22:21.662 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:22:24.194 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-04-25T07:22:24.194 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-04-25T07:22:24.194 INFO:tasks.workunit.client.0.smithi088.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-04-25T07:22:24.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:22:25.509 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:22:25.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-04-25T07:22:26.110 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records in 2024-04-25T07:22:26.110 INFO:tasks.workunit.client.0.smithi088.stderr:1024+0 records out 2024-04-25T07:22:26.110 INFO:tasks.workunit.client.0.smithi088.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.582241 s, 1.8 GB/s 2024-04-25T07:22:26.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-04-25T07:22:26.134 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:22:34.418 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-04-25T07:22:34.418 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-04-25T07:22:34.418 INFO:tasks.workunit.client.0.smithi088.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-04-25T07:22:34.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:22:35.734 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:22:35.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:22:35.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:22:35.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:22:35.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:22:35.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:22:35.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:22:35.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:22:35.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:22:35.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:35.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:35.753 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:35.753 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:35.753 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:22:35.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:22:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:22:35.756 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:22:35.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:22:35.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:22:35.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:22:35.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:22:35.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:22:35.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:22:35.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:22:35.790 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:35.791+0000 7f52339d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:35.793 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:35.791+0000 7f52339d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:35.794 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:35.795+0000 7f52339d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:22:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:22:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:22:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:22:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:22:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:22:36.163 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:22:36.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:36.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:36.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:36.836 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:36.835+0000 7f52339d77c0 -1 Falling back to public interface 2024-04-25T07:22:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:22:37.575 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:22:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:37.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:37.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:38.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:38.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:38.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:22:38.983 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:22:38.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:38.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:39.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:40.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:40.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:40.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:22:40.383 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:22:40.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:40.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:40.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:41.660 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:41.659+0000 7f52339d77c0 -1 osd.0 75 log_to_monitors true 2024-04-25T07:22:41.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:41.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:41.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:22:41.787 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:22:41.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:41.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:42.324 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 86 up_thru 86 down_at 76 last_clean_interval [61,75) [v2:127.0.0.1:6802/332440236,v1:127.0.0.1:6803/332440236] [v2:127.0.0.1:6804/332440236,v1:127.0.0.1:6805/332440236] exists,up 85e4cd7e-efc8-49c2-a5ee-9fe41f509b63 2024-04-25T07:22:42.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:22:42.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:22:42.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:22:42.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:22:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=137307 2024-04-25T07:22:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:22:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:22:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:22:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:22:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:22:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:22:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:22:42.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:22:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:22:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:22:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:22:42.331 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:22:42.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:22:42.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:22:42.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:22:42.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:22:42.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:22:42.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:22:42.363 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:42.363+0000 7f17bd8907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:42.378 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:42.379+0000 7f17bd8907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:42.379 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:42.379+0000 7f17bd8907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:42.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:22:42.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:22:42.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:22:42.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:22:42.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:22:42.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:42.746 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:22:42.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:22:42.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:42.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:42.923 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:42.923+0000 7f17bd8907c0 -1 Falling back to public interface 2024-04-25T07:22:43.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:44.193 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:22:44.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:22:44.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:44.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:44.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:45.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:22:45.606 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:22:45.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:45.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:46.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:46.393 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:46.391+0000 7f17bd8907c0 -1 osd.1 78 log_to_monitors true 2024-04-25T07:22:47.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:47.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:47.016 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:22:47.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:22:47.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:47.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:47.420 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 79 last_clean_interval [64,78) [v2:127.0.0.1:6810/2453001106,v1:127.0.0.1:6811/2453001106] [v2:127.0.0.1:6812/2453001106,v1:127.0.0.1:6813/2453001106] exists,up 1dd411e3-2838-4aa5-96cc-e48db2d17c8a 2024-04-25T07:22:47.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:22:47.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:22:47.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:22:47.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=138276 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:47.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:22:47.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:22:47.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:22:47.428 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:22:47.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:22:47.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:22:47.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:22:47.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:22:47.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:22:47.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:22:47.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:22:47.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:47.451+0000 7f02e8e627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:47.596 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:47.595+0000 7f02e8e627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:47.598 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:47.599+0000 7f02e8e627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:47.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:22:47.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:22:47.946 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:22:47.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:22:47.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:22:47.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:22:47.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:47.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:22:47.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:47.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:22:48.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:48.404 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:48.403+0000 7f02e8e627c0 -1 Falling back to public interface 2024-04-25T07:22:49.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:49.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:49.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:22:49.354 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:22:49.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:49.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:22:49.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:50.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:50.790 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:22:50.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:50.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:22:50.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:22:50.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:51.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:52.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:52.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:52.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:22:52.250 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:22:52.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:52.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:22:52.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:52.411+0000 7f02e8e627c0 -1 osd.2 81 log_to_monitors true 2024-04-25T07:22:52.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:53.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:53.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:53.768 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:22:53.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:22:53.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:53.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:22:54.336 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 93 up_thru 93 down_at 82 last_clean_interval [68,81) [v2:127.0.0.1:6818/1369912464,v1:127.0.0.1:6819/1369912464] [v2:127.0.0.1:6820/1369912464,v1:127.0.0.1:6821/1369912464] exists,up ff8c4f8b-f5b7-4cce-a6f2-10031e4d44b6 2024-04-25T07:22:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:22:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:22:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:22:54.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:22:54.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=139224 2024-04-25T07:22:54.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:22:54.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:22:54.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:22:54.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:54.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:54.342 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:54.342 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:54.342 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:22:54.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:22:54.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:22:54.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:54.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:22:54.343 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T07:22:54.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:22:54.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:22:54.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:22:54.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:22:54.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:22:54.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:22:54.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:22:54.372 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:54.370+0000 7f0e07d697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:54.392 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:54.390+0000 7f0e07d697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:54.394 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:54.394+0000 7f0e07d697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:54.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:22:54.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:22:54.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:22:54.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:22:54.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:22:54.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:54.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:22:54.753 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:22:54.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:54.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:22:55.184 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:55.182+0000 7f0e07d697c0 -1 Falling back to public interface 2024-04-25T07:22:55.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:56.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:56.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:56.207 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:22:56.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:22:56.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:56.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:22:56.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:57.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:57.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:57.633 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:22:57.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:22:57.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:57.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:22:58.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:59.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:59.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:59.058 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:22:59.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:22:59.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:59.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:22:59.206 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:22:59.206+0000 7f0e07d697c0 -1 osd.3 83 log_to_monitors true 2024-04-25T07:22:59.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:00.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:00.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:00.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:23:00.606 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:23:00.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:00.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:23:01.130 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 98 up_thru 98 down_at 84 last_clean_interval [73,83) [v2:127.0.0.1:6826/3220255551,v1:127.0.0.1:6827/3220255551] [v2:127.0.0.1:6828/3220255551,v1:127.0.0.1:6829/3220255551] exists,up d8a7062b-bbee-43c1-b1ec-73d5c8fbbcd5 2024-04-25T07:23:01.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:23:01.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:23:01.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:23:01.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:23:01.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=140174 2024-04-25T07:23:01.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:23:01.168 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T07:23:01.168 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:23:01.168 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_141123 2024-04-25T07:23:01.168 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:23:01.168 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:23:02.168 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 104 88 0.343729 0.34375 0.196385 0.171243 2024-04-25T07:23:03.168 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 208 192 0.374961 0.40625 0.181936 0.160151 2024-04-25T07:23:04.168 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 320 304 0.395791 0.4375 0.189265 0.153615 2024-04-25T07:23:05.168 INFO:tasks.workunit.client.0.smithi088.stdout: 4 16 433 417 0.407185 0.441406 0.099362 0.151096 2024-04-25T07:23:06.168 INFO:tasks.workunit.client.0.smithi088.stdout: 5 16 532 516 0.40308 0.386719 0.194985 0.150972 2024-04-25T07:23:07.169 INFO:tasks.workunit.client.0.smithi088.stdout: 6 16 641 625 0.406854 0.425781 0.108128 0.151354 2024-04-25T07:23:08.169 INFO:tasks.workunit.client.0.smithi088.stdout: 7 16 753 737 0.411223 0.4375 0.156883 0.1495 2024-04-25T07:23:09.169 INFO:tasks.workunit.client.0.smithi088.stdout: 8 15 875 860 0.419872 0.480469 0.121888 0.148245 2024-04-25T07:23:10.169 INFO:tasks.workunit.client.0.smithi088.stdout: 9 16 981 965 0.418787 0.410156 0.183561 0.148138 2024-04-25T07:23:11.169 INFO:tasks.workunit.client.0.smithi088.stdout: 10 16 1061 1045 0.408153 0.3125 0.160414 0.151683 2024-04-25T07:23:12.169 INFO:tasks.workunit.client.0.smithi088.stdout: 11 16 1182 1166 0.414013 0.472656 0.12202 0.149986 2024-04-25T07:23:13.169 INFO:tasks.workunit.client.0.smithi088.stdout: 12 16 1299 1283 0.417592 0.457031 0.155283 0.148764 2024-04-25T07:23:14.169 INFO:tasks.workunit.client.0.smithi088.stdout: 13 16 1398 1382 0.415215 0.386719 0.186069 0.14884 2024-04-25T07:23:15.169 INFO:tasks.workunit.client.0.smithi088.stdout: 14 16 1504 1488 0.415128 0.414062 0.121501 0.149309 2024-04-25T07:23:16.170 INFO:tasks.workunit.client.0.smithi088.stdout: 15 16 1596 1580 0.411409 0.359375 0.270285 0.150787 2024-04-25T07:23:17.170 INFO:tasks.workunit.client.0.smithi088.stdout: 16 16 1680 1664 0.406202 0.328125 0.142641 0.152909 2024-04-25T07:23:18.170 INFO:tasks.workunit.client.0.smithi088.stdout: 17 16 1789 1773 0.407351 0.425781 0.150565 0.152381 2024-04-25T07:23:19.170 INFO:tasks.workunit.client.0.smithi088.stdout: 18 16 1883 1867 0.405117 0.367188 0.177554 0.153913 2024-04-25T07:23:20.170 INFO:tasks.workunit.client.0.smithi088.stdout: 19 16 1971 1955 0.401886 0.34375 0.1583 0.154607 2024-04-25T07:23:21.170 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:23:21.174652+0000 min lat: 0.077959 max lat: 0.470666 avg lat: 0.154801 2024-04-25T07:23:21.170 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:23:21.170 INFO:tasks.workunit.client.0.smithi088.stdout: 20 16 2079 2063 0.402883 0.421875 0.152668 0.154801 2024-04-25T07:23:22.170 INFO:tasks.workunit.client.0.smithi088.stdout: 21 16 2161 2145 0.398949 0.320312 0.148779 0.15562 2024-04-25T07:23:23.170 INFO:tasks.workunit.client.0.smithi088.stdout: 22 16 2280 2264 0.401943 0.464844 0.100298 0.155095 2024-04-25T07:23:24.174 INFO:tasks.workunit.client.0.smithi088.stdout: 23 16 2390 2374 0.403147 0.429688 0.172041 0.154541 2024-04-25T07:23:25.170 INFO:tasks.workunit.client.0.smithi088.stdout: 24 16 2495 2479 0.403437 0.410156 0.12761 0.154483 2024-04-25T07:23:26.171 INFO:tasks.workunit.client.0.smithi088.stdout: 25 16 2603 2587 0.404173 0.421875 0.192563 0.153848 2024-04-25T07:23:27.171 INFO:tasks.workunit.client.0.smithi088.stdout: 26 16 2682 2666 0.400495 0.308594 0.325717 0.155068 2024-04-25T07:23:28.171 INFO:tasks.workunit.client.0.smithi088.stdout: 27 16 2791 2775 0.40143 0.425781 0.149385 0.155144 2024-04-25T07:23:29.171 INFO:tasks.workunit.client.0.smithi088.stdout: 28 16 2903 2887 0.402717 0.4375 0.160667 0.154832 2024-04-25T07:23:30.171 INFO:tasks.workunit.client.0.smithi088.stdout: 29 16 3008 2992 0.402972 0.410156 0.16805 0.154551 2024-04-25T07:23:31.171 INFO:tasks.workunit.client.0.smithi088.stdout: 30 16 3114 3098 0.40334 0.414062 0.126528 0.154656 2024-04-25T07:23:32.171 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 30.0796 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 3114 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4096 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4096 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.404395 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.0478189 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.480469 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.308594 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 103 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 12.2416 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 123 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 79 2024-04-25T07:23:32.172 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.154411 2024-04-25T07:23:32.173 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0440954 2024-04-25T07:23:32.173 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.470666 2024-04-25T07:23:32.173 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.077959 2024-04-25T07:23:32.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-04-25T07:23:32.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:23:32.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:23:32.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:23:32.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:23:32.258 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:23:32.258 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:23:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:23:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:23:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:23:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:23:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:23:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:23:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:23:32.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:23:32.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:23:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:23:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:23:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:23:32.806 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T07:23:32.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:23:32.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:32.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:23:32.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=369367187469 2024-04-25T07:23:32.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 369367187469 2024-04-25T07:23:32.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-369367187469' 2024-04-25T07:23:32.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:32.978 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:23:33.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=382252089356 2024-04-25T07:23:33.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 382252089356 2024-04-25T07:23:33.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-369367187469 1-382252089356' 2024-04-25T07:23:33.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:33.132 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:23:33.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=399431958539 2024-04-25T07:23:33.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 399431958539 2024-04-25T07:23:33.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958539' 2024-04-25T07:23:33.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:33.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:23:33.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=420906795017 2024-04-25T07:23:33.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 420906795017 2024-04-25T07:23:33.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958539 3-420906795017' 2024-04-25T07:23:33.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:33.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-369367187469 2024-04-25T07:23:33.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:33.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:23:33.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-369367187469 2024-04-25T07:23:33.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:33.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=369367187469 2024-04-25T07:23:33.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 369367187469' 2024-04-25T07:23:33.441 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 369367187469 2024-04-25T07:23:33.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:23:33.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 369367187469 -lt 369367187469 2024-04-25T07:23:33.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:33.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:33.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-382252089356 2024-04-25T07:23:33.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:23:33.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-382252089356 2024-04-25T07:23:33.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:33.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=382252089356 2024-04-25T07:23:33.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 382252089356' 2024-04-25T07:23:33.842 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 382252089356 2024-04-25T07:23:33.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:23:34.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 382252089356 -lt 382252089356 2024-04-25T07:23:34.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:34.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:34.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-399431958539 2024-04-25T07:23:34.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:23:34.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-399431958539 2024-04-25T07:23:34.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:34.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=399431958539 2024-04-25T07:23:34.279 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 399431958539 2024-04-25T07:23:34.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 399431958539' 2024-04-25T07:23:34.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:23:34.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 399431958539 -lt 399431958539 2024-04-25T07:23:34.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:34.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-420906795017 2024-04-25T07:23:34.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:34.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:23:34.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-420906795017 2024-04-25T07:23:34.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:34.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=420906795017 2024-04-25T07:23:34.665 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 420906795017 2024-04-25T07:23:34.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 420906795017' 2024-04-25T07:23:34.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:23:35.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 420906795017 -lt 420906795017 2024-04-25T07:23:35.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:23:35.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:35.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:35.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:23:35.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:23:35.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:23:35.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:23:35.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:23:35.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:23:35.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:23:35.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:23:36.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:23:36.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:23:36.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:36.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:23:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:23:36.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:23:36.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:23:36.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:23:36.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:23:36.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:23:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:23:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:23:36.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:23:36.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:23:36.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:23:36.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:23:36.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:23:36.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:23:36.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:23:36.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:23:37.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:23:37.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:37.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:23:37.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94617 2024-04-25T07:23:37.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:23:37.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:23:37.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:23:37.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:23:37.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:23:37.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:23:37.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:23:37.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:23:37.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:23:37.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:23:37.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:23:37.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:23:37.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:23:37.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:23:37.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:23:37.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:23:37.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:23:37.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:23:37.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:23:37.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:23:37.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:23:37.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:23:37.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:23:37.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:23:37.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:23:37.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:23:37.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:23:37.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:23:37.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:23:37.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:37.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:23:37.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94617 2024-04-25T07:23:37.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:23:37.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:23:37.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:23:37.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:23:37.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:37.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:23:37.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94617 2024-04-25T07:23:37.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:23:37.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:23:37.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:23:37.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:23:37.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-04-25T07:23:37.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:23:37.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-04-25T07:23:37.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-04-25T07:23:37.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-04-25T07:23:37.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:23:37.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-04-25T07:23:37.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-04-25T07:23:37.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-04-25T07:23:37.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:23:37.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:23:37.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:23:37.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:23:37.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:23:37.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:23:37.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:23:37.868 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:23:37.868 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:37.868 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:23:37.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:23:37.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:23:38.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:23:38.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:23:38.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:23:38.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:23:38.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:23:38.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:23:38.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:23:38.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:23:38.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:23:38.081 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:23:38.082 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:38.082 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:23:38.082 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94617/ceph-mon.a.asok 2024-04-25T07:23:38.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:23:38.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94617/ceph-mon.a.asok config get fsid 2024-04-25T07:23:38.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:23:38.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:23:38.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:23:38.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:23:38.201 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:23:38.201 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:23:38.201 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:23:38.202 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:23:38.202 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:23:38.202 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:38.202 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:23:38.202 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94617/ceph-mon.a.asok 2024-04-25T07:23:38.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:23:38.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94617/ceph-mon.a.asok config get mon_host 2024-04-25T07:23:38.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:23:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:23:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:23:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:23:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:23:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:23:38.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:23:39.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:23:39.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:23:39.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:23:39.218 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:23:39.218 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:39.218 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:23:39.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:23:39.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:23:39.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:23:39.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:23:39.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:23:39.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:23:39.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:23:39.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:23:39.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:23:39.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:23:39.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:23:39.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:23:39.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:23:39.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:23:39.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a0caed9-8f93-48b1-b528-85ec94228d1d 2024-04-25T07:23:39.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2a0caed9-8f93-48b1-b528-85ec94228d1d' 2024-04-25T07:23:39.261 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 2a0caed9-8f93-48b1-b528-85ec94228d1d 2024-04-25T07:23:39.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:23:39.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7BCpmbqzqEBAAfLE4KMiOqWeDDLgaMU6cvQ== 2024-04-25T07:23:39.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7BCpmbqzqEBAAfLE4KMiOqWeDDLgaMU6cvQ=="}' 2024-04-25T07:23:39.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a0caed9-8f93-48b1-b528-85ec94228d1d -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:23:39.564 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:23:39.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:23:39.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7BCpmbqzqEBAAfLE4KMiOqWeDDLgaMU6cvQ== --osd-uuid 2a0caed9-8f93-48b1-b528-85ec94228d1d 2024-04-25T07:23:39.618 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:39.621+0000 7f90ff0b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:39.633 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:39.633+0000 7f90ff0b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:39.634 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:39.637+0000 7f90ff0b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:39.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:39.637+0000 7f90ff0b97c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:23:39.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:39.637+0000 7f90ff0b97c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:23:39.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:39.637+0000 7f90ff0b97c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:23:39.673 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:39.673+0000 7f90ff0b97c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:23:47.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:23:47.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:23:47.993 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:23:47.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:23:47.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:23:48.572 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:23:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:23:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:23:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:23:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:23:48.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:23:48.609 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:48.609+0000 7f087b7357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:48.646 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:48.649+0000 7f087b7357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:48.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:48.649+0000 7f087b7357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:49.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:23:49.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:23:49.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:23:49.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:23:49.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:23:49.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:49.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:23:49.012 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:23:49.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:23:49.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:49.448 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:49.449+0000 7f087b7357c0 -1 Falling back to public interface 2024-04-25T07:23:49.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:50.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:50.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:50.470 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:23:50.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:23:50.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:50.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:23:50.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:51.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:51.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:51.886 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:23:51.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:23:51.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:51.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:23:52.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:53.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:53.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:53.316 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:23:53.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:23:53.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:53.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:23:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:23:53.485+0000 7f087b7357c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:23:53.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:54.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:54.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:54.735 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:23:54.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:23:54.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:54.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:23:55.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:56.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:56.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:56.767 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:23:56.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:23:56.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:56.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:23:57.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:58.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:58.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:58.855 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:23:58.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:23:58.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:58.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:23:59.278 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4085900055,v1:127.0.0.1:6803/4085900055] [v2:127.0.0.1:6804/4085900055,v1:127.0.0.1:6805/4085900055] exists,up 2a0caed9-8f93-48b1-b528-85ec94228d1d 2024-04-25T07:23:59.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:23:59.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:23:59.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:23:59.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:23:59.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=142995 2024-04-25T07:23:59.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-04-25T07:24:04.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-04-25T07:24:04.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:24:04.832 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T07:24:04.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:24:05.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-04-25T07:24:05.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-04-25T07:24:05.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-04-25T07:24:05.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-04-25T07:24:05.962 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T07:24:05.964 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-04-25T07:24:05.964 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_144276 2024-04-25T07:24:06.000 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:24:06.001 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:24:07.001 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 17 1 3.99955 4 0.206186 0.206186 2024-04-25T07:24:08.001 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 33 17 33.9953 64 1.03544 1.07878 2024-04-25T07:24:09.001 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 49 33 43.9935 64 0.90164 0.990178 2024-04-25T07:24:10.001 INFO:tasks.workunit.client.0.smithi088.stdout: 4 16 80 64 63.9913 124 0.884436 0.974504 2024-04-25T07:24:11.001 INFO:tasks.workunit.client.0.smithi088.stdout: 5 16 81 65 51.9929 4 0.926651 0.973768 2024-04-25T07:24:12.001 INFO:tasks.workunit.client.0.smithi088.stdout: 6 16 97 81 53.9925 64 1.39505 1.06259 2024-04-25T07:24:13.002 INFO:tasks.workunit.client.0.smithi088.stdout: 7 16 113 97 55.4212 64 1.19491 1.08335 2024-04-25T07:24:14.002 INFO:tasks.workunit.client.0.smithi088.stdout: 8 16 129 113 56.4924 64 0.934712 1.05852 2024-04-25T07:24:15.002 INFO:tasks.workunit.client.0.smithi088.stdout: 9 16 145 129 57.3256 64 1.26167 1.08156 2024-04-25T07:24:16.002 INFO:tasks.workunit.client.0.smithi088.stdout: 10 16 161 145 57.992 64 0.767861 1.05309 2024-04-25T07:24:17.002 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 10.7883 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 161 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4194304 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4194304 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 59.6944 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 34.0588 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 124 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 4 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 14 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 8.51469 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 31 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 1 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 1.06047 2024-04-25T07:24:17.003 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.194132 2024-04-25T07:24:17.004 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 1.44113 2024-04-25T07:24:17.004 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.206186 2024-04-25T07:24:17.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-04-25T07:24:27.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:24:27.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-04-25T07:24:27.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1059061760 2024-04-25T07:24:27.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:24:27.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-04-25T07:24:27.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=92405760 2024-04-25T07:24:27.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-04-25T07:24:27.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 92405760 -eq 0 2024-04-25T07:24:27.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-04-25T07:24:27.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 92405760 -gt 0 2024-04-25T07:24:27.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 142995 2024-04-25T07:24:27.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-04-25T07:24:28.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 142995 2024-04-25T07:24:28.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (142995) - No such process 2024-04-25T07:24:28.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-04-25T07:24:28.888 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T07:24:28.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-04-25T07:24:28.934 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:24:38.474 INFO:tasks.workunit.client.0.smithi088.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-04-25T07:24:38.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-04-25T07:24:38.526 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:24:40.311 INFO:tasks.workunit.client.0.smithi088.stdout:1 : device size 0xffffe000 : using 0x44c37000(1.1 GiB) 2024-04-25T07:24:40.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-04-25T07:24:42.390 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:24:42.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:24:42.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:24:42.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:24:42.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:24:42.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:24:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:24:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:24:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:24:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:24:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:24:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:24:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:24:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:24:42.438 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:24:42.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:24:42.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:24:42.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:24:42.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:24:42.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:24:42.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:24:42.474 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:24:42.476+0000 7f92a64e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:42.477 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:24:42.476+0000 7f92a64e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:42.478 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:24:42.480+0000 7f92a64e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:42.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:24:42.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:24:42.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:24:42.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:24:42.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:24:42.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:42.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:24:42.858 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:24:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:42.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:43.023 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:24:43.024+0000 7f92a64e47c0 -1 Falling back to public interface 2024-04-25T07:24:43.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:44.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:44.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:44.313 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:24:44.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:24:44.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:44.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:44.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:45.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:45.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:45.729 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:24:45.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:24:45.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:45.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:46.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:47.177 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:24:47.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:24:47.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:47.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:47.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:48.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:48.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:48.590 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:24:48.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:24:48.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:48.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:49.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:50.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:50.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:50.021 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:24:50.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:24:50.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:50.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:50.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:51.364 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:24:51.363+0000 7f92a64e47c0 -1 osd.0 10 log_to_monitors true 2024-04-25T07:24:51.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:51.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:51.436 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:24:51.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:24:51.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:51.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:52.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:53.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:53.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:53.144 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:24:53.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:24:53.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:53.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:53.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:54.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:54.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:54.787 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:24:54.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:24:54.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:54.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:55.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:56.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:56.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:56.339 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:24:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:24:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:56.854 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/3783603671,v1:127.0.0.1:6803/3783603671] [v2:127.0.0.1:6804/3783603671,v1:127.0.0.1:6805/3783603671] exists,up 2a0caed9-8f93-48b1-b528-85ec94228d1d 2024-04-25T07:24:56.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:24:56.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:24:56.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:24:56.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:24:56.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=145501 2024-04-25T07:24:56.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-04-25T07:24:56.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:24:56.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:24:56.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:24:56.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:24:56.857 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:24:56.857 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:24:56.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:24:56.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:24:56.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:24:56.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:24:56.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:24:56.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:24:56.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:24:56.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:24:56.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:24:57.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T07:24:57.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:24:57.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:57.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:24:57.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574852 2024-04-25T07:24:57.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574852 2024-04-25T07:24:57.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574852' 2024-04-25T07:24:57.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574852 2024-04-25T07:24:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:57.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:24:57.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-04-25T07:24:57.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:57.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574852 2024-04-25T07:24:57.571 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 55834574852 2024-04-25T07:24:57.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-04-25T07:24:57.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:24:58.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574852 -lt 55834574852 2024-04-25T07:24:58.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:24:58.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:24:58.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:24:58.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:24:58.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:24:58.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:24:58.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:24:58.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:24:58.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:24:58.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:24:58.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:24:59.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:24:59.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:24:59.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:24:59.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:24:59.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:24:59.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:24:59.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:24:59.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:24:59.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:24:59.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:24:59.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:24:59.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:24:59.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:24:59.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:24:59.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:24:59.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:24:59.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:24:59.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:24:59.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:24:59.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:24:59.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:24:59.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:24:59.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:24:59.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:59.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:24:59.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:24:59.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:59.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:24:59.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:24:59.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:24:59.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:24:59.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:59.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:24:59.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94617 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:24:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:24:59.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:24:59.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:24:59.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:24:59.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:24:59.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:24:59.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:24:59.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:24:59.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:24:59.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:24:59.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:59.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:24:59.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:24:59.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:24:59.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:24:59.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:24:59.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:24:59.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:24:59.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:59.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:24:59.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94617 2024-04-25T07:24:59.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:24:59.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:24:59.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:24:59.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:24:59.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:59.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:24:59.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.94617 2024-04-25T07:24:59.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:24:59.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:24:59.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:24:59.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:24:59.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-04-25T07:24:59.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:24:59.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-04-25T07:24:59.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-04-25T07:24:59.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-04-25T07:24:59.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:24:59.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-04-25T07:24:59.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-04-25T07:24:59.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-04-25T07:24:59.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=30ec9a36-34cf-422f-8146-a6090ddee72a --auth-supported=none ' 2024-04-25T07:24:59.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:24:59.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-04-25T07:24:59.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:24:59.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:24:59.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-04-25T07:24:59.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:24:59.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:25:00.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:25:00.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:00.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:00.273 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:00.274 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:00.274 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:25:00.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:25:00.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:25:00.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:25:00.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:25:00.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:25:00.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:25:00.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:25:00.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:25:00.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:25:00.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:25:00.513 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:25:00.513 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:25:00.513 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:00.513 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:25:00.513 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94617/ceph-mon.a.asok 2024-04-25T07:25:00.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:25:00.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94617/ceph-mon.a.asok config get fsid 2024-04-25T07:25:00.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:25:00.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:25:00.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:25:00.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:25:00.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:25:00.626 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:25:00.626 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:25:00.626 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:25:00.627 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:25:00.627 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:00.627 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:25:00.627 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.94617/ceph-mon.a.asok 2024-04-25T07:25:00.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:25:00.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.94617/ceph-mon.a.asok config get mon_host 2024-04-25T07:25:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:25:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:25:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:25:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:25:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:25:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:25:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:25:01.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:25:01.646 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:01.646 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:01.646 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:01.646 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:01.646 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:25:01.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:25:01.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:25:01.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=30ec9a36-34cf-422f-8146-a6090ddee72a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:25:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:25:01.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:25:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:25:01.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:25:01.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fbd593f1-61c6-484e-b20a-9708ee358744 2024-04-25T07:25:01.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fbd593f1-61c6-484e-b20a-9708ee358744' 2024-04-25T07:25:01.693 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 fbd593f1-61c6-484e-b20a-9708ee358744 2024-04-25T07:25:01.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:25:01.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNBSpmSyjiKhAACbngh1PcGOFbQ+ek8vBf5g== 2024-04-25T07:25:01.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNBSpmSyjiKhAACbngh1PcGOFbQ+ek8vBf5g=="}' 2024-04-25T07:25:01.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fbd593f1-61c6-484e-b20a-9708ee358744 -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:25:02.047 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:25:02.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:25:02.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=30ec9a36-34cf-422f-8146-a6090ddee72a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNBSpmSyjiKhAACbngh1PcGOFbQ+ek8vBf5g== --osd-uuid fbd593f1-61c6-484e-b20a-9708ee358744 2024-04-25T07:25:02.102 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:02.103+0000 7efccd8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:02.115 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:02.115+0000 7efccd8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:02.116 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:02.115+0000 7efccd8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:02.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:02.115+0000 7efccd8087c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:25:02.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:02.115+0000 7efccd8087c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:25:02.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:02.115+0000 7efccd8087c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:25:02.155 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:02.155+0000 7efccd8087c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:25:10.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:25:10.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:25:10.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:25:10.246 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:25:10.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:25:10.816 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:25:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:25:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=30ec9a36-34cf-422f-8146-a6090ddee72a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:25:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:25:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:25:10.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:25:10.859 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:10.859+0000 7fb6bdbed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:10.862 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:10.863+0000 7fb6bdbed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:10.863 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:10.863+0000 7fb6bdbed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:11.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:25:11.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:25:11.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:25:11.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:25:11.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:25:11.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:11.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:25:11.295 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:25:11.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:11.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:11.669 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:11.667+0000 7fb6bdbed7c0 -1 Falling back to public interface 2024-04-25T07:25:11.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:12.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:12.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:12.728 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:25:12.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:25:12.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:12.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:13.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:14.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:14.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:14.167 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:25:14.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:25:14.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:14.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:14.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:15.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:15.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:15.575 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:25:15.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:25:15.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:15.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:15.667 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:25:15.667+0000 7fb6bdbed7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:25:16.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:17.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:17.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:17.471 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:25:17.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:25:17.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:17.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:18.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:19.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:19.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:19.929 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:25:19.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:25:19.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:19.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:20.405 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1230411533,v1:127.0.0.1:6803/1230411533] [v2:127.0.0.1:6804/1230411533,v1:127.0.0.1:6805/1230411533] exists,up fbd593f1-61c6-484e-b20a-9708ee358744 2024-04-25T07:25:20.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:25:20.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:25:20.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:25:20.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:25:20.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=148248 2024-04-25T07:25:20.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-04-25T07:25:25.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-04-25T07:25:25.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:25:25.932 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T07:25:25.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:25:26.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:25:27.062 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T07:25:27.062 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:25:27.062 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_149249 2024-04-25T07:25:27.062 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:25:27.062 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:25:28.062 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 128 112 0.43745 0.4375 0.10069 0.133562 2024-04-25T07:25:29.062 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 241 225 0.439387 0.441406 0.160317 0.137272 2024-04-25T07:25:30.063 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 385 369 0.480405 0.5625 0.108841 0.125777 2024-04-25T07:25:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: 4 16 513 497 0.48529 0.5 0.100452 0.126808 2024-04-25T07:25:32.063 INFO:tasks.workunit.client.0.smithi088.stdout: 5 16 656 640 0.499938 0.558594 0.132933 0.123838 2024-04-25T07:25:33.063 INFO:tasks.workunit.client.0.smithi088.stdout: 6 16 785 769 0.500591 0.503906 0.0921146 0.124104 2024-04-25T07:25:34.063 INFO:tasks.workunit.client.0.smithi088.stdout: 7 16 928 912 0.508865 0.558594 0.100718 0.121717 2024-04-25T07:25:35.063 INFO:tasks.workunit.client.0.smithi088.stdout: 8 16 1041 1025 0.500423 0.441406 0.169074 0.123592 2024-04-25T07:25:36.063 INFO:tasks.workunit.client.0.smithi088.stdout: 9 16 1137 1121 0.486483 0.375 0.134012 0.12721 2024-04-25T07:25:37.063 INFO:tasks.workunit.client.0.smithi088.stdout: 10 16 1264 1248 0.48744 0.496094 0.159214 0.127908 2024-04-25T07:25:38.063 INFO:tasks.workunit.client.0.smithi088.stdout: 11 16 1408 1392 0.494256 0.5625 0.0999917 0.125662 2024-04-25T07:25:39.064 INFO:tasks.workunit.client.0.smithi088.stdout: 12 16 1521 1505 0.489848 0.441406 0.176024 0.126632 2024-04-25T07:25:40.064 INFO:tasks.workunit.client.0.smithi088.stdout: 13 16 1664 1648 0.49513 0.558594 0.101011 0.125967 2024-04-25T07:25:41.064 INFO:tasks.workunit.client.0.smithi088.stdout: 14 16 1776 1760 0.491009 0.4375 0.14173 0.127077 2024-04-25T07:25:42.064 INFO:tasks.workunit.client.0.smithi088.stdout: 15 16 1905 1889 0.491863 0.503906 0.0921824 0.126676 2024-04-25T07:25:43.064 INFO:tasks.workunit.client.0.smithi088.stdout: 16 16 2017 2001 0.488463 0.4375 0.184197 0.127029 2024-04-25T07:25:44.064 INFO:tasks.workunit.client.0.smithi088.stdout: 17 16 2160 2144 0.492583 0.558594 0.100384 0.126616 2024-04-25T07:25:45.065 INFO:tasks.workunit.client.0.smithi088.stdout: 18 16 2273 2257 0.489736 0.441406 0.125525 0.127204 2024-04-25T07:25:46.065 INFO:tasks.workunit.client.0.smithi088.stdout: 19 16 2400 2384 0.490066 0.496094 0.101443 0.127405 2024-04-25T07:25:47.065 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:25:47.068659+0000 min lat: 0.0901945 max lat: 0.358944 avg lat: 0.127635 2024-04-25T07:25:47.065 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:25:47.065 INFO:tasks.workunit.client.0.smithi088.stdout: 20 16 2513 2497 0.487631 0.441406 0.142413 0.127635 2024-04-25T07:25:48.065 INFO:tasks.workunit.client.0.smithi088.stdout: 21 16 2641 2625 0.488216 0.5 0.109062 0.127531 2024-04-25T07:25:49.065 INFO:tasks.workunit.client.0.smithi088.stdout: 22 16 2769 2753 0.488748 0.5 0.142459 0.127391 2024-04-25T07:25:50.065 INFO:tasks.workunit.client.0.smithi088.stdout: 23 16 2912 2896 0.491782 0.558594 0.116978 0.126847 2024-04-25T07:25:51.065 INFO:tasks.workunit.client.0.smithi088.stdout: 24 16 3041 3025 0.492285 0.503906 0.100402 0.126781 2024-04-25T07:25:52.066 INFO:tasks.workunit.client.0.smithi088.stdout: 25 16 3184 3168 0.494934 0.558594 0.101352 0.12623 2024-04-25T07:25:53.066 INFO:tasks.workunit.client.0.smithi088.stdout: 26 16 3281 3265 0.490469 0.378906 0.219526 0.126651 2024-04-25T07:25:54.066 INFO:tasks.workunit.client.0.smithi088.stdout: 27 16 3440 3424 0.495303 0.621094 0.109702 0.126098 2024-04-25T07:25:55.066 INFO:tasks.workunit.client.0.smithi088.stdout: 28 16 3552 3536 0.493237 0.4375 0.173985 0.126647 2024-04-25T07:25:56.066 INFO:tasks.workunit.client.0.smithi088.stdout: 29 16 3681 3665 0.493602 0.503906 0.108734 0.126314 2024-04-25T07:25:57.066 INFO:tasks.workunit.client.0.smithi088.stdout: 30 16 3808 3792 0.493683 0.496094 0.109192 0.126537 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 30.129 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 3808 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4096 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4096 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.493711 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.060198 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.621094 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.375 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 126 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 15.4107 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 159 2024-04-25T07:25:58.067 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 96 2024-04-25T07:25:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.126567 2024-04-25T07:25:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0305919 2024-04-25T07:25:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.358944 2024-04-25T07:25:58.068 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.0901945 2024-04-25T07:25:58.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-04-25T07:26:03.116 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-04-25T07:26:03.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-04-25T07:26:03.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-04-25T07:26:03.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-04-25T07:26:03.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-04-25T07:26:03.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-04-25T07:26:03.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-04-25T07:26:03.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-04-25T07:26:03.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-04-25T07:26:03.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4279267328 2024-04-25T07:26:03.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 148248 2024-04-25T07:26:03.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-04-25T07:26:04.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 148248 2024-04-25T07:26:04.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (148248) - No such process 2024-04-25T07:26:04.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-04-25T07:26:04.996 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T07:26:05.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-04-25T07:26:05.046 INFO:tasks.workunit.client.0.smithi088.stdout:allocmap bluestore.allocmap 2024-04-25T07:26:08.079 INFO:tasks.workunit.client.0.smithi088.stdout:allocmap success 2024-04-25T07:26:08.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:446: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:26:11.159 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:26:11.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:448: TEST_bluestore_expand: requested_space=4294967296 2024-04-25T07:26:11.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:449: TEST_bluestore_expand: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-04-25T07:26:11.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:450: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-04-25T07:26:11.224 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:26:13.289 INFO:tasks.workunit.client.0.smithi088.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-04-25T07:26:13.289 INFO:tasks.workunit.client.0.smithi088.stdout:2 : device size 0x100000000 : using 0xef9000(15 MiB) 2024-04-25T07:26:13.289 INFO:tasks.workunit.client.0.smithi088.stdout:Expanding DB/WAL... 2024-04-25T07:26:13.289 INFO:tasks.workunit.client.0.smithi088.stdout:1 : expanding to 0x1073741824 2024-04-25T07:26:13.313 INFO:tasks.workunit.client.0.smithi088.stdout:1 : size label updated to 1073741824 2024-04-25T07:26:14.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:453: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:26:16.850 INFO:tasks.workunit.client.0.smithi088.stdout:fsck success 2024-04-25T07:26:16.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:456: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-04-25T07:26:16.905 INFO:tasks.workunit.client.0.smithi088.stdout:qfsck bluestore.quick-fsck 2024-04-25T07:26:19.687 INFO:tasks.workunit.client.0.smithi088.stdout:qfsck success 2024-04-25T07:26:19.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:458: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-04-25T07:26:19.736 INFO:tasks.workunit.client.0.smithi088.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:26:22.520 INFO:tasks.workunit.client.0.smithi088.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-04-25T07:26:22.520 INFO:tasks.workunit.client.0.smithi088.stdout:2 : device size 0x100000000 : using 0xef9000(15 MiB) 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:460: TEST_bluestore_expand: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=30ec9a36-34cf-422f-8146-a6090ddee72a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:22.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:22.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:26:22.533 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:26:22.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:26:22.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=21cf5e85-6d16-4926-9a90-98c5c681fdca --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=3ce1073a-7e55-4800-912f-3daf0cd2d302 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=30ec9a36-34cf-422f-8146-a6090ddee72a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.94617/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:22.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:26:22.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:26:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:22.577 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:22.577+0000 7f2a6178f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:22.580 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:22.581+0000 7f2a6178f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:22.581 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:22.581+0000 7f2a6178f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:22.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:26:22.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:22.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:26:22.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:22.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:22.970 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:26:22.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:22.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:22.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:22.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:23.364 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:23.365+0000 7f2a6178f7c0 -1 Falling back to public interface 2024-04-25T07:26:23.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:26:24.368 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:26:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:24.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:25.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:25.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:25.807 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:26:25.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:26:25.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:25.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:26.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:27.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:27.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:27.252 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:26:27.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:26:27.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:27.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:27.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:28.374 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:28.373+0000 7f2a6178f7c0 -1 osd.0 10 log_to_monitors true 2024-04-25T07:26:28.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:28.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:28.704 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:26:28.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:26:28.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:28.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:30.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:31.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:31.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:31.043 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:26:31.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:26:31.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:31.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:31.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:32.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:32.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:32.515 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:26:32.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:26:32.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:32.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:33.212 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/3001371866,v1:127.0.0.1:6803/3001371866] [v2:127.0.0.1:6804/3001371866,v1:127.0.0.1:6805/3001371866] exists,up fbd593f1-61c6-484e-b20a-9708ee358744 2024-04-25T07:26:33.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:33.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:33.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:33.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:26:33.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: osd_pid0=151600 2024-04-25T07:26:33.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:463: TEST_bluestore_expand: wait_for_clean 2024-04-25T07:26:33.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:26:33.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:26:33.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:26:33.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:26:33.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:26:33.215 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:26:33.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:26:33.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:26:33.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:26:33.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:26:33.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:26:33.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:26:33.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:26:33.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:26:33.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:26:33.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T07:26:33.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:26:33.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:26:33.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:26:33.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574852 2024-04-25T07:26:33.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574852 2024-04-25T07:26:33.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574852' 2024-04-25T07:26:33.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:26:33.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574852 2024-04-25T07:26:33.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:26:33.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:26:33.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-04-25T07:26:33.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:26:33.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574852 2024-04-25T07:26:33.944 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 55834574852 2024-04-25T07:26:33.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-04-25T07:26:33.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:26:34.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574852 -lt 55834574852 2024-04-25T07:26:34.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:26:34.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:34.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:34.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:26:34.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:34.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:34.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:34.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:34.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:34.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:34.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:35.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:26:35.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:35.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:35.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:35.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:26:36.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:26:36.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:26:36.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-04-25T07:26:36.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-04-25T07:26:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: total_space_after=4294967296 2024-04-25T07:26:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-04-25T07:26:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-04-25T07:26:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep free 2024-04-25T07:26:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d: -f 2 2024-04-25T07:26:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d, -f 1 2024-04-25T07:26:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut '-d ' -f 2 2024-04-25T07:26:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: free_space_after=4279222272 2024-04-25T07:26:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:468: TEST_bluestore_expand: '[4294967296' '!=' '4294967296]' 2024-04-25T07:26:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 468: [4294967296: command not found 2024-04-25T07:26:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:474: TEST_bluestore_expand: total_space_added=0 2024-04-25T07:26:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:475: TEST_bluestore_expand: free_space_added=-45056 2024-04-25T07:26:36.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:477: TEST_bluestore_expand: let 'new_used_space=(0 - -45056)' 2024-04-25T07:26:36.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:478: TEST_bluestore_expand: echo 45056 2024-04-25T07:26:36.251 INFO:tasks.workunit.client.0.smithi088.stdout:45056 2024-04-25T07:26:36.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:480: TEST_bluestore_expand: '[' 45056 -gt 131072 ']' 2024-04-25T07:26:36.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 151600 2024-04-25T07:26:36.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2024-04-25T07:26:37.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 151600 2024-04-25T07:26:37.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 487: kill: (151600) - No such process 2024-04-25T07:26:37.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:488: TEST_bluestore_expand: ceph osd down 0 2024-04-25T07:26:37.831 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T07:26:37.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:490: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-04-25T07:26:37.883 INFO:tasks.workunit.client.0.smithi088.stdout:qfsck bluestore.quick-fsck 2024-04-25T07:26:40.415 INFO:tasks.workunit.client.0.smithi088.stdout:qfsck success 2024-04-25T07:26:40.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:26:40.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:26:40.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:26:40.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:26:40.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:40.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:40.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:40.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:40.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:40.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:40.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:40.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:40.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:40.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:40.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:40.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:40.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:40.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:40.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:40.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:40.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:40.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:26:40.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:26:40.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:40.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:40.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:26:40.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94617 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-bluefs-volume-ops 0 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:40.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:40.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:40.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:40.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:40.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:40.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:40.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:40.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:40.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:40.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:40.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:40.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:40.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:40.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:26:40.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:26:40.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:40.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:40.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.94617 2024-04-25T07:26:40.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.94617 2024-04-25T07:26:40.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:40.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:40.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:26:40.853 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:26:40.854 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:26:40.906 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2024-04-25T07:26:40.906 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2024-04-25T07:26:40.958 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:26:40.963 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:26:40.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:26:40.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-config 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2024-04-25T07:26:40.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2024-04-25T07:26:40.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=4599658a-1a56-4c66-9bc5-3eab33dd38e9 --auth-supported=none ' 2024-04-25T07:26:40.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-04-25T07:26:40.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2024-04-25T07:26:40.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_config_track 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_default_adjustment' 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:40.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:40.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:40.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:40.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:40.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:40.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:40.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:40.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:40.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:40.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:40.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:40.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:40.974 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:40.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:40.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:26:40.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:26:40.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:40.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:40.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:26:40.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153298 2024-04-25T07:26:40.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:40.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:40.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:26:40.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:26:40.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:40.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:26:40.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153298 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:26:40.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-04-25T07:26:40.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-04-25T07:26:41.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:26:41.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:41.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:41.452 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:41.452 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:41.452 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:26:41.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153298/$cluster-$name.asok' 2024-04-25T07:26:41.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:26:41.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:26:41.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:26:41.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:26:41.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:26:41.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:26:41.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:26:41.680 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:26:41.680 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:26:41.680 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:26:41.681 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:41.681 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:41.681 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:26:41.681 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153298/ceph-mon.a.asok 2024-04-25T07:26:41.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:26:41.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153298/ceph-mon.a.asok config get fsid 2024-04-25T07:26:41.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:26:41.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:26:41.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:26:41.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:26:41.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:26:41.798 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:26:41.798 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:26:41.798 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:26:41.799 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:41.799 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:41.799 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:26:41.799 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153298/ceph-mon.a.asok 2024-04-25T07:26:41.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:26:41.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153298/ceph-mon.a.asok config get mon_host 2024-04-25T07:26:41.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2024-04-25T07:26:41.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-04-25T07:26:41.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:26:41.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:26:41.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:26:41.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-04-25T07:26:41.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:26:42.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:26:42.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:42.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:42.809 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:42.809 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:42.809 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:26:42.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153298/$cluster-$name.asok' 2024-04-25T07:26:42.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:26:42.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4599658a-1a56-4c66-9bc5-3eab33dd38e9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T07:26:42.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:26:42.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153298/$cluster-$name.asok' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-04-25T07:26:42.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:26:42.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a16dae6-8092-4572-ac82-171013c49d68 2024-04-25T07:26:42.848 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 8a16dae6-8092-4572-ac82-171013c49d68 2024-04-25T07:26:42.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8a16dae6-8092-4572-ac82-171013c49d68' 2024-04-25T07:26:42.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:26:42.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyBSpmv7aLMxAAus5qPfrAwjuFYilxJCSpZw== 2024-04-25T07:26:42.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyBSpmv7aLMxAAus5qPfrAwjuFYilxJCSpZw=="}' 2024-04-25T07:26:42.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a16dae6-8092-4572-ac82-171013c49d68 -i td/osd-config/0/new.json 2024-04-25T07:26:43.149 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:26:43.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-04-25T07:26:43.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4599658a-1a56-4c66-9bc5-3eab33dd38e9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQCyBSpmv7aLMxAAus5qPfrAwjuFYilxJCSpZw== --osd-uuid 8a16dae6-8092-4572-ac82-171013c49d68 2024-04-25T07:26:43.198 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:43.197+0000 7f0d1dff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:43.218 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:43.217+0000 7f0d1dff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:43.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:43.221+0000 7f0d1dff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:43.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:43.221+0000 7f0d1dff57c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:26:43.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:43.221+0000 7f0d1dff57c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:26:43.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:43.221+0000 7f0d1dff57c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:26:43.250 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:43.249+0000 7f0d1dff57c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-04-25T07:26:49.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-04-25T07:26:49.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:26:49.588 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:26:49.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:26:49.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:26:50.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:26:50.144 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:26:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4599658a-1a56-4c66-9bc5-3eab33dd38e9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-04-25T07:26:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:26:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:26:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:26:50.169 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:50.169+0000 7f3cccca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:50.172 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:50.173+0000 7f3cccca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:50.173 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:50.173+0000 7f3cccca47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:50.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:26:50.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:50.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:26:50.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:50.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:50.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:50.561 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:26:50.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:50.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:50.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:50.740 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:50.741+0000 7f3cccca47c0 -1 Falling back to public interface 2024-04-25T07:26:50.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:51.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:51.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:51.963 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:26:51.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:26:51.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:51.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:52.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:26:53.321+0000 7f3cccca47c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:26:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:26:53.361 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:26:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:54.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:55.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:55.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:55.046 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:26:55.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:26:55.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:55.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:55.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:56.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:56.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:56.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:26:56.495 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:26:56.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:56.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:56.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:57.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:57.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:57.944 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:26:57.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:26:57.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:57.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:26:59.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:26:59.505 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:26:59.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:59.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:26:59.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:59.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:59.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:00.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:00.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:00.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:27:00.943 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:27:00.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:00.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:01.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:02.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:02.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:02.481 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:27:02.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:27:02.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:02.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:02.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:03.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:03.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:03.909 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:27:03.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:27:03.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:03.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:04.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:05.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:05.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:05.388 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:27:05.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:27:05.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:05.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:05.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:06.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:06.816 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:27:06.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:27:06.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:06.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:07.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:08.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:08.768 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:27:08.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:08.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:27:08.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:08.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:09.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:10.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:10.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:10.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:27:10.536 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:27:10.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:10.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:11.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:12.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:12.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:12.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:27:12.195 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:27:12.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:12.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:13.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:14.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:14.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:14.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:27:14.518 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:27:14.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:14.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:15.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:16.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:16.272 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:27:16.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:16.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:27:16.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:16.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:17.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:18.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:18.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:18.155 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:27:18.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:27:18.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:18.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:18.597 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1905025581,v1:127.0.0.1:6803/1905025581] [v2:127.0.0.1:6804/1905025581,v1:127.0.0.1:6805/1905025581] exists,up 8a16dae6-8092-4572-ac82-171013c49d68 2024-04-25T07:27:18.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:27:18.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:27:18.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:27:18.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2024-04-25T07:27:18.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:27:18.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:27:18.600 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:18.600 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:18.600 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:18.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153298/ceph-osd.0.asok 2024-04-25T07:27:18.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2024-04-25T07:27:18.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.153298/ceph-osd.0.asok log flush 2024-04-25T07:27:18.696 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:27:18.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:27:18.710 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:26:53.697+0000 7f3cccca47c0 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:27:18.711 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:26:56.320+0000 7f3cbcc46640 10 log_client will send 2024-04-25T07:26:53.700112+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:27:18.711 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:26:56.472+0000 7f3cbdc48640 10 log_client logged 2024-04-25T07:26:53.700112+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:27:18.711 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:27:17.508+0000 7f3cc845d640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:27:18.711 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:27:18.324+0000 7f3cbcc46640 10 log_client will send 2024-04-25T07:27:17.512410+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:27:18.711 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:27:18.432+0000 7f3cbdc48640 10 log_client logged 2024-04-25T07:27:17.512410+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:27:18.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:27:18.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:27:18.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:27:18.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:27:18.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:18.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:18.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:18.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:18.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:19.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:19.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:19.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:19.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:19.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:19.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:19.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:19.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:19.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:19.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:19.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:19.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:19.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:27:19.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:27:19.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:19.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:19.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:19.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153298 2024-04-25T07:27:19.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:19.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:19.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:27:19.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:27:19.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:19.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:19.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:19.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:19.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:19.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:19.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:19.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:19.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:19.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:27:19.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:27:19.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:19.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:19.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:19.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153298 2024-04-25T07:27:19.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:19.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:19.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:27:19.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:27:19.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:19.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:19.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153298 2024-04-25T07:27:19.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:27:19.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:27:19.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:27:19.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:27:19.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2024-04-25T07:27:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2024-04-25T07:27:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2024-04-25T07:27:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-04-25T07:27:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:27:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:27:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:27:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-04-25T07:27:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-04-25T07:27:19.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:27:19.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:19.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:19.547 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:19.547 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:19.547 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:19.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153298/$cluster-$name.asok' 2024-04-25T07:27:19.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:27:19.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:27:19.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:27:19.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:27:19.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:27:19.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:27:19.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:27:19.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:27:19.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:27:19.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:27:19.743 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:19.744 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:19.744 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:19.744 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153298/ceph-mon.a.asok 2024-04-25T07:27:19.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:27:19.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153298/ceph-mon.a.asok config get fsid 2024-04-25T07:27:19.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:27:19.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:27:19.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:27:19.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:27:19.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:27:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:27:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:27:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:27:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:19.862 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153298/ceph-mon.a.asok 2024-04-25T07:27:19.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:27:19.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.153298/ceph-mon.a.asok config get mon_host 2024-04-25T07:27:19.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2024-04-25T07:27:19.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-04-25T07:27:19.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:27:19.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:27:19.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:27:19.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-04-25T07:27:19.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:27:20.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:27:20.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:20.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:20.890 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:20.890 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:20.890 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:20.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153298/$cluster-$name.asok' 2024-04-25T07:27:20.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:27:20.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:27:20.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2024-04-25T07:27:20.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4599658a-1a56-4c66-9bc5-3eab33dd38e9 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:27:20.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.153298/$cluster-$name.asok' 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:27:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-04-25T07:27:20.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:27:20.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8cc4ea82-af4b-4534-b0e5-b3f1aa17cc7c 2024-04-25T07:27:20.928 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 8cc4ea82-af4b-4534-b0e5-b3f1aa17cc7c 2024-04-25T07:27:20.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8cc4ea82-af4b-4534-b0e5-b3f1aa17cc7c' 2024-04-25T07:27:20.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:27:20.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYBSpmp4tLOBAApVdRzlJnz8Z8b58A9HvKsA== 2024-04-25T07:27:20.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYBSpmp4tLOBAApVdRzlJnz8Z8b58A9HvKsA=="}' 2024-04-25T07:27:20.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8cc4ea82-af4b-4534-b0e5-b3f1aa17cc7c -i td/osd-config/0/new.json 2024-04-25T07:27:21.260 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:27:21.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-04-25T07:27:21.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4599658a-1a56-4c66-9bc5-3eab33dd38e9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYBSpmp4tLOBAApVdRzlJnz8Z8b58A9HvKsA== --osd-uuid 8cc4ea82-af4b-4534-b0e5-b3f1aa17cc7c 2024-04-25T07:27:21.308 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:21.308+0000 7f751b7487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:21.338 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:21.336+0000 7f751b7487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:21.339 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:21.340+0000 7f751b7487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:21.340 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:21.340+0000 7f751b7487c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:27:21.340 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:21.340+0000 7f751b7487c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:27:21.340 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:21.340+0000 7f751b7487c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:27:21.369 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:21.368+0000 7f751b7487c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-04-25T07:27:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-04-25T07:27:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:27:27.735 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:27:27.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:27:27.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:27:28.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:27:28.287 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:27:28.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4599658a-1a56-4c66-9bc5-3eab33dd38e9 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.153298/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:27:28.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:27:28.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:27:28.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:27:28.316 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:28.316+0000 7fbfaac787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:28.319 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:28.320+0000 7fbfaac787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:28.320 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:28.320+0000 7fbfaac787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:27:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:27:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:28.692 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:27:28.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:28.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:28.896 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:28.896+0000 7fbfaac787c0 -1 Falling back to public interface 2024-04-25T07:27:29.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:30.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:30.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:30.108 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:27:30.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:27:30.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:30.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:30.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:31.427 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:27:31.428+0000 7fbfaac787c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:27:31.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:31.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:31.510 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:27:31.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:27:31.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:31.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:32.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:33.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:33.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:33.106 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:27:33.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:27:33.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:33.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:33.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:34.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:34.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:34.576 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:27:34.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:27:34.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:34.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:35.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:36.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:36.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:36.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:27:36.022 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:27:36.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:36.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:36.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:37.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:37.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:37.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:27:37.612 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:27:37.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:37.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:38.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:39.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:39.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:39.041 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:27:39.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:27:39.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:39.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:39.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:40.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:40.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:40.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:27:40.604 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:27:40.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:40.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:41.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:42.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:42.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:42.016 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:27:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:27:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:42.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:43.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:43.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:43.578 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:27:43.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:27:43.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:43.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:44.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:45.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:45.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:45.259 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:27:45.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:27:45.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:45.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:46.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:47.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:47.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:47.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:27:47.158 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:27:47.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:47.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:47.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:48.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:48.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:48.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:27:48.617 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:27:48.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:48.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:49.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:50.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:50.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:50.363 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:27:50.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:27:50.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:50.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:51.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:52.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:52.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:52.536 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:27:52.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:27:52.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:52.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:53.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:54.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:54.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:54.368 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:27:54.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:27:54.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:54.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:55.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:56.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:56.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:56.272 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:27:56.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:27:56.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:56.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:56.715 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/929909986,v1:127.0.0.1:6803/929909986] [v2:127.0.0.1:6804/929909986,v1:127.0.0.1:6805/929909986] exists,up 8cc4ea82-af4b-4534-b0e5-b3f1aa17cc7c 2024-04-25T07:27:56.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:27:56.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:27:56.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:27:56.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2024-04-25T07:27:56.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2024-04-25T07:27:56.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2024-04-25T07:27:56.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2024-04-25T07:27:56.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2024-04-25T07:27:56.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2024-04-25T07:27:56.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:27:56.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2024-04-25T07:27:56.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2024-04-25T07:27:56.888 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:27:56.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2024-04-25T07:27:56.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:27:56.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:27:56.904 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:56.904 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:56.904 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:56.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153298/ceph-osd.0.asok 2024-04-25T07:27:56.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2024-04-25T07:27:56.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.153298/ceph-osd.0.asok log flush 2024-04-25T07:27:57.008 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:27:57.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:27:57.023 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T07:27:56.875+0000 7fbfa9437640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2024-04-25T07:27:57.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2024-04-25T07:27:57.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2024-04-25T07:27:57.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:27:57.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:27:57.025 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:57.025 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:57.025 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:57.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.153298/ceph-osd.0.asok 2024-04-25T07:27:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2024-04-25T07:27:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.153298/ceph-osd.0.asok log reopen 2024-04-25T07:27:57.122 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:27:57.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:27:57.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:27:57.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:27:57.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:27:57.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:57.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:57.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:57.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:57.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:57.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:57.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:57.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:57.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:57.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:57.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:57.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:57.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:57.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:57.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:27:57.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:27:57.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:57.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:57.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:57.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153298 2024-04-25T07:27:57.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:57.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:57.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:27:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:27:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:27:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:27:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:27:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:27:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:27:57.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:57.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:57.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:57.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:57.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:57.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:57.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:57.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:57.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:57.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:57.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:57.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:57.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:57.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:57.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:57.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:27:57.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:27:57.476 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:57.476 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:57.476 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:57.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153298 2024-04-25T07:27:57.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:57.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:57.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:27:57.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:27:57.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:57.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:57.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.153298 2024-04-25T07:27:57.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:27:57.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:27:57.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:27:57.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:27:57.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2024-04-25T07:27:57.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2024-04-25T07:27:57.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2024-04-25T07:27:57.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2024-04-25T07:27:57.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2024-04-25T07:27:57.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2024-04-25T07:27:57.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2024-04-25T07:27:57.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2024-04-25T07:27:57.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2024-04-25T07:27:57.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2024-04-25T07:27:57.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2024-04-25T07:27:57.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2024-04-25T07:27:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2024-04-25T07:27:57.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2024-04-25T07:27:57.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:57.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:57.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:57.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:57.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:57.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:57.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:57.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:57.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:57.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:57.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:57.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:57.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:57.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:27:57.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:27:57.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:57.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:57.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:57.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153298 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-config 0 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:57.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:57.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:57.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:57.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:57.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:57.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:57.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:57.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:57.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:57.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:57.630 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:57.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:27:57.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:27:57.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:57.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:57.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.153298 2024-04-25T07:27:57.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.153298 2024-04-25T07:27:57.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:57.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:57.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:27:57.636 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:27:57.636 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:27:57.690 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2024-04-25T07:27:57.690 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2024-04-25T07:27:57.747 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:27:57.761 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:27:57.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:27:57.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:27:57.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:27:57.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:27:57.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:27:57.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:27:57.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:27:57.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:27:57.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-copy-from 2024-04-25T07:27:57.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2024-04-25T07:27:57.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2024-04-25T07:27:57.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=3597c642-5268-4c61-b759-aa6517455aad --auth-supported=none ' 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:27:57.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:57.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:57.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.157414 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:57.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2024-04-25T07:27:57.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:27:57.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:57.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:27:57.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.157414 2024-04-25T07:27:57.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:27:57.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2024-04-25T07:27:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2024-04-25T07:27:58.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:27:58.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:58.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:58.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:58.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:58.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:27:58.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.157414/$cluster-$name.asok' 2024-04-25T07:27:58.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.157414/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:27:58.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:27:58.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:27:58.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:27:58.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:27:58.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:27:58.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:27:58.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:27:58.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:27:58.473 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:27:58.473 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:58.473 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:58.473 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:27:58.474 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.157414/ceph-mon.a.asok 2024-04-25T07:27:58.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:27:58.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.157414/ceph-mon.a.asok config get fsid 2024-04-25T07:27:58.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:27:58.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:27:58.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:27:58.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:27:58.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:27:58.591 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:27:58.591 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:27:58.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:27:58.592 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:58.593 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:58.593 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:27:58.593 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.157414/ceph-mon.a.asok 2024-04-25T07:27:58.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:27:58.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.157414/ceph-mon.a.asok config get mon_host 2024-04-25T07:27:58.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2024-04-25T07:27:58.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2024-04-25T07:27:58.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:27:58.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:27:58.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:27:58.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2024-04-25T07:27:58.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:27:59.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:27:59.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:59.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:59.623 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:59.623 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:59.623 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:27:59.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.157414/$cluster-$name.asok' 2024-04-25T07:27:59.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:27:59.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.157414/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:27:59.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2024-04-25T07:27:59.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-04-25T07:27:59.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:27:59.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:27:59.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3597c642-5268-4c61-b759-aa6517455aad --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:59.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.157414/$cluster-$name.asok' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.157414/$cluster-$name.asok' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-04-25T07:27:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:27:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:27:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:27:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:27:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:27:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:27:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2024-04-25T07:27:59.665 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:27:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=300d26f4-005f-45bb-a94a-3f9a766efcdc 2024-04-25T07:27:59.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 300d26f4-005f-45bb-a94a-3f9a766efcdc' 2024-04-25T07:27:59.666 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 300d26f4-005f-45bb-a94a-3f9a766efcdc 2024-04-25T07:27:59.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:27:59.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/BSpmx7KzKBAAOo2K2OBDd3LJ6ew8TaM4fQ== 2024-04-25T07:27:59.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/BSpmx7KzKBAAOo2K2OBDd3LJ6ew8TaM4fQ=="}' 2024-04-25T07:27:59.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 300d26f4-005f-45bb-a94a-3f9a766efcdc -i td/osd-copy-from/0/new.json 2024-04-25T07:27:59.973 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:27:59.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2024-04-25T07:27:59.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3597c642-5268-4c61-b759-aa6517455aad --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.157414/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/BSpmx7KzKBAAOo2K2OBDd3LJ6ew8TaM4fQ== --osd-uuid 300d26f4-005f-45bb-a94a-3f9a766efcdc 2024-04-25T07:28:00.014 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:00.011+0000 7fc04fe7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:00.033 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:00.031+0000 7fc04fe7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:00.034 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:00.031+0000 7fc04fe7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:00.034 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:00.031+0000 7fc04fe7b7c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:28:00.035 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:00.031+0000 7fc04fe7b7c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:28:00.035 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:00.031+0000 7fc04fe7b7c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:28:00.073 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:00.071+0000 7fc04fe7b7c0 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2024-04-25T07:28:06.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2024-04-25T07:28:06.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:28:06.277 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:28:06.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:28:06.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:28:06.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:28:06.827 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:28:06.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3597c642-5268-4c61-b759-aa6517455aad --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.157414/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:28:06.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:28:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:28:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:28:06.856 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:06.855+0000 7fca9cb347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:06.887 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:06.887+0000 7fca9cb347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:06.889 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:06.887+0000 7fca9cb347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:07.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:28:07.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:28:07.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:28:07.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:28:07.248 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:28:07.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:28:07.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:07.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:28:07.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:07.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:07.445 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:07.443+0000 7fca9cb347c0 -1 Falling back to public interface 2024-04-25T07:28:07.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:08.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:08.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:08.671 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:28:08.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:28:08.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:08.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:09.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:09.998 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:09.995+0000 7fca9cb347c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:28:10.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:10.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:10.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:28:10.073 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:28:10.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:10.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:10.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:11.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:11.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:11.720 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:28:11.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:28:11.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:11.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:12.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:13.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:13.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:13.160 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:28:13.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:28:13.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:13.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:13.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:14.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:14.598 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:28:14.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:28:14.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:14.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:15.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:16.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:16.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:16.159 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:28:16.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:28:16.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:16.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:16.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:17.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:17.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:17.591 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:28:17.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:28:17.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:17.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:18.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:19.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:19.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:19.116 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:28:19.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:28:19.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:19.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:19.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:20.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:20.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:20.563 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:28:20.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:28:20.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:20.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:21.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:22.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:22.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:22.033 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:28:22.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:28:22.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:22.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:22.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:23.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:23.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:23.671 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:28:23.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:28:23.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:23.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:24.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:25.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:25.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:25.366 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:28:25.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:28:25.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:25.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:26.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:27.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:27.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:27.160 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:28:27.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:28:27.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:27.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:27.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:28.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:28.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:28.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:28:28.919 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:28:28.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:28.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:30.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:31.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:31.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:31.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:28:31.067 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:28:31.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:31.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:31.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:32.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:32.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:32.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:28:32.803 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:28:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:32.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:33.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:34.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:34.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:34.494 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:28:34.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:28:34.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:34.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:34.959 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3323036200,v1:127.0.0.1:6803/3323036200] [v2:127.0.0.1:6804/3323036200,v1:127.0.0.1:6805/3323036200] exists,up 300d26f4-005f-45bb-a94a-3f9a766efcdc 2024-04-25T07:28:34.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:28:34.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:28:34.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:28:34.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3597c642-5268-4c61-b759-aa6517455aad --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:28:34.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:28:34.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-04-25T07:28:34.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:28:34.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:28:34.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:28:34.963 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:28:34.963 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:34.963 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.157414/$cluster-$name.asok' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.157414/$cluster-$name.asok' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:28:34.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:28:34.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:28:34.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:28:34.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:28:34.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2024-04-25T07:28:34.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:28:34.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=719676c5-0c91-4ad7-91dd-59c9c043a167 2024-04-25T07:28:34.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 719676c5-0c91-4ad7-91dd-59c9c043a167' 2024-04-25T07:28:34.966 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 719676c5-0c91-4ad7-91dd-59c9c043a167 2024-04-25T07:28:34.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:28:34.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiBipmRKjlOhAAVFMft8AY8lRLUuH+FZcu5Q== 2024-04-25T07:28:34.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiBipmRKjlOhAAVFMft8AY8lRLUuH+FZcu5Q=="}' 2024-04-25T07:28:34.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 719676c5-0c91-4ad7-91dd-59c9c043a167 -i td/osd-copy-from/1/new.json 2024-04-25T07:28:35.441 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:28:35.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2024-04-25T07:28:35.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3597c642-5268-4c61-b759-aa6517455aad --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.157414/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiBipmRKjlOhAAVFMft8AY8lRLUuH+FZcu5Q== --osd-uuid 719676c5-0c91-4ad7-91dd-59c9c043a167 2024-04-25T07:28:35.488 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:35.486+0000 7fad4583d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:35.490 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:35.490+0000 7fad4583d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:35.492 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:35.490+0000 7fad4583d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:35.493 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:35.490+0000 7fad4583d7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:28:35.493 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:35.490+0000 7fad4583d7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:28:35.493 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:35.490+0000 7fad4583d7c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:28:35.576 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:35.574+0000 7fad4583d7c0 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2024-04-25T07:28:42.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2024-04-25T07:28:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:28:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:28:42.328 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:28:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:28:42.880 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:28:42.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:28:42.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3597c642-5268-4c61-b759-aa6517455aad --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.157414/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:28:42.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:28:42.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:28:42.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:28:42.910 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:42.906+0000 7f9828d117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:42.913 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:42.910+0000 7f9828d117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:42.914 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:42.910+0000 7f9828d117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:43.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:28:43.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:28:43.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:28:43.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:28:43.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:28:43.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:43.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:28:43.289 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:28:43.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:43.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:43.488 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:43.486+0000 7f9828d117c0 -1 Falling back to public interface 2024-04-25T07:28:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:44.695 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:28:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:28:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:45.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:46.121 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:28:46.118+0000 7f9828d117c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:28:46.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:46.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:46.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:28:46.126 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:28:46.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:46.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:46.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:47.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:47.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:47.581 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:28:47.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:28:47.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:47.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:48.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:49.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:49.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:49.101 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:28:49.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:28:49.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:49.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:49.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:50.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:50.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:50.735 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:28:50.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:28:50.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:50.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:51.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:52.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:52.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:52.148 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:28:52.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:28:52.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:52.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:52.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:53.565 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:28:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:28:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:54.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:55.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:55.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:55.068 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:28:55.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:28:55.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:55.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:55.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:56.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:56.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:56.501 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:28:56.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:28:56.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:56.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:57.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:58.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:58.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:58.045 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:28:58.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:28:58.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:58.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:58.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:59.532 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:28:59.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:28:59.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:59.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:00.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:01.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:01.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:01.182 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:29:01.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:29:01.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:01.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:01.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:02.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:02.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:02.912 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:29:02.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:29:02.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:02.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:03.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:04.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:04.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:04.786 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:29:04.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:29:04.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:04.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:05.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:06.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:06.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:06.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:29:06.713 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:29:06.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:06.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:07.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:08.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:08.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:08.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:29:08.534 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:29:08.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:08.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:09.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:10.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:10.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:10.435 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:29:10.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:29:10.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:10.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:11.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:12.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:12.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:12.324 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T07:29:12.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:29:12.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:12.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:12.751 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2194875726,v1:127.0.0.1:6811/2194875726] [v2:127.0.0.1:6812/2194875726,v1:127.0.0.1:6813/2194875726] exists,up 719676c5-0c91-4ad7-91dd-59c9c043a167 2024-04-25T07:29:12.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:12.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:12.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:12.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2024-04-25T07:29:12.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:29:13.196 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-04-25T07:29:13.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:29:13.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:29:13.737 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-04-25T07:29:13.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:29:14.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:29:16.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2024-04-25T07:29:16.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2024-04-25T07:29:16.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2024-04-25T07:29:17.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2024-04-25T07:29:17.055 INFO:tasks.workunit.client.0.smithi088.stdout:rbd/foo2 mtime 2024-04-25T07:29:16.000000+0000, size 572320 2024-04-25T07:29:17.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2024-04-25T07:29:17.196 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T07:29:17.228 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T07:29:17.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2024-04-25T07:29:17.279 INFO:tasks.workunit.client.0.smithi088.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2024-04-25T07:29:17.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2024-04-25T07:29:17.326 INFO:tasks.workunit.client.0.smithi088.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2024-04-25T07:29:17.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2024-04-25T07:29:17.464 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T07:29:17.493 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T07:29:17.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2024-04-25T07:29:17.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2024-04-25T07:29:17.749 INFO:tasks.workunit.client.0.smithi088.stdout:rbd/foo3 mtime 2024-04-25T07:29:16.000000+0000, size 572320 2024-04-25T07:29:17.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2024-04-25T07:29:17.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:29:17.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:29:17.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:29:17.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:17.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:17.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:17.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:17.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:18.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:18.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:29:18.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:29:18.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:29:18.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:29:18.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:29:18.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:29:18.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:18.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:29:18.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:29:18.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:18.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:29:18.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:29:18.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:29:18.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:29:18.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:18.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:29:18.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.157414 2024-04-25T07:29:18.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:29:18.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:29:18.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:29:18.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-copy-from 0 2024-04-25T07:29:18.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:29:18.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:29:18.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:29:18.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:18.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:18.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:18.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:18.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:18.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:18.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:29:18.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:29:18.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:29:18.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:29:18.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:29:18.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:29:18.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:18.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:29:18.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:29:18.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:18.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:29:18.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:29:18.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:29:18.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:29:18.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:18.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.157414 2024-04-25T07:29:18.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.157414 2024-04-25T07:29:18.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:29:18.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:29:18.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:29:18.100 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:29:18.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:29:18.158 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2024-04-25T07:29:18.159 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2024-04-25T07:29:18.215 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:29:18.221 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:29:18.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:29:18.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:29:18.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-dup 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2024-04-25T07:29:18.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2024-04-25T07:29:18.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2024-04-25T07:29:18.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=99751126-f7fc-4bfb-a600-7a2083496976 --auth-supported=none ' 2024-04-25T07:29:18.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:29:18.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2024-04-25T07:29:18.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-04-25T07:29:18.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2024-04-25T07:29:18.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:29:18.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs= 2024-04-25T07:29:18.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:29:18.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-dup 0 2024-04-25T07:29:18.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-04-25T07:29:18.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:29:18.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-04-25T07:29:18.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:18.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:18.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:18.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:18.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:18.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:18.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:29:18.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:29:18.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:29:18.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:29:18.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:29:18.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:29:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:29:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:29:18.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:18.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:29:18.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:29:18.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-04-25T07:29:18.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:29:18.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:18.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161474 2024-04-25T07:29:18.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161474 2024-04-25T07:29:18.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:29:18.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:29:18.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:29:18.238 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:29:18.238 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:29:18.289 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2024-04-25T07:29:18.289 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2024-04-25T07:29:18.343 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:29:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:29:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:29:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:29:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:29:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:29:18.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:29:18.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:29:18.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:29:18.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:29:18.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-fast-mark-down 2024-04-25T07:29:18.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2024-04-25T07:29:18.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2024-04-25T07:29:18.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2024-04-25T07:29:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2024-04-25T07:29:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2024-04-25T07:29:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2024-04-25T07:29:18.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2024-04-25T07:29:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none ' 2024-04-25T07:29:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none ' 2024-04-25T07:29:18.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2024-04-25T07:29:18.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2024-04-25T07:29:18.351 INFO:tasks.workunit.client.0.smithi088.stdout:Ensuring old behavior is there... 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:18.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:18.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:18.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:18.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:29:18.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:29:18.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:29:18.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:29:18.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:29:18.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:29:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:29:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:29:18.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:18.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:29:18.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:29:18.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:29:18.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:29:18.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:18.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:18.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161503 2024-04-25T07:29:18.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:29:18.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:29:18.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-04-25T07:29:18.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:29:18.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:18.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:18.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.161503 2024-04-25T07:29:18.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:29:18.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:29:18.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:29:18.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:29:18.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-04-25T07:29:18.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-04-25T07:29:18.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-04-25T07:29:18.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:29:18.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:29:18.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:29:18.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-04-25T07:29:18.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-04-25T07:29:18.397 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:18.398+0000 7f0f4f83bc80 -1 failed for service _ceph-mon._tcp 2024-04-25T07:29:18.397 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:18.398+0000 7f0f4f83bc80 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-25T07:29:18.397 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T07:29:18.397 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:18.398+0000 7f0f4f83bc80 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2024-04-25T07:29:18.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:29:18.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:18.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:18.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:18.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:18.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:18.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:29:18.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T07:29:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:19.046+0000 7f5208160c80 -1 failed for service _ceph-mon._tcp 2024-04-25T07:29:19.045 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:19.046+0000 7f5208160c80 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-25T07:29:19.045 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T07:29:19.045 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:19.046+0000 7f5208160c80 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2024-04-25T07:29:19.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2024-04-25T07:29:19.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2024-04-25T07:29:19.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-04-25T07:29:19.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-04-25T07:29:19.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:29:19.054 INFO:tasks.workunit.client.0.smithi088.stdout:Testing async msgr... 2024-04-25T07:29:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2024-04-25T07:29:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2024-04-25T07:29:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-04-25T07:29:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-04-25T07:29:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-04-25T07:29:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:29:19.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:29:19.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:29:19.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:19.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:19.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:19.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:19.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:19.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:19.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:29:19.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:29:19.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:29:19.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:29:19.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:29:19.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:29:19.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:19.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:29:19.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:29:19.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:19.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:29:19.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:29:19.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:29:19.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:29:19.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:19.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:19.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161503 2024-04-25T07:29:19.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:29:19.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:29:19.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-04-25T07:29:19.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:29:19.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:19.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:19.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.161503 2024-04-25T07:29:19.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:29:19.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:29:19.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:29:19.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-04-25T07:29:19.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-04-25T07:29:19.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-04-25T07:29:19.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:29:19.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:29:19.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:29:19.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-04-25T07:29:19.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-04-25T07:29:19.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:29:19.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:19.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:19.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:29:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T07:29:19.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:29:19.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:29:19.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:29:19.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:29:19.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:29:19.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:29:19.781 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:29:19.782 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:29:19.782 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:29:19.782 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:29:19.782 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:19.782 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:19.782 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.161503/ceph-mon.a.asok 2024-04-25T07:29:19.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:29:19.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.161503/ceph-mon.a.asok config get fsid 2024-04-25T07:29:19.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:29:19.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:29:19.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:29:19.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.161503/ceph-mon.a.asok 2024-04-25T07:29:19.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:29:19.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.161503/ceph-mon.a.asok config get mon_host 2024-04-25T07:29:20.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2024-04-25T07:29:20.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2024-04-25T07:29:20.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:29:20.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:29:20.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:29:20.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2024-04-25T07:29:20.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:29:20.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:29:20.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:20.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:20.916 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:20.916 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:20.916 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:20.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:29:20.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:29:20.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:29:20.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:29:20.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:29:20.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:20.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:20.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:20.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:20.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:29:20.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:29:20.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2024-04-25T07:29:20.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:29:20.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=77b1f0bf-91e8-4654-964f-157876cf2a2c 2024-04-25T07:29:20.963 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 77b1f0bf-91e8-4654-964f-157876cf2a2c 2024-04-25T07:29:20.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 77b1f0bf-91e8-4654-964f-157876cf2a2c' 2024-04-25T07:29:20.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:29:20.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQBipmO5+POhAABeqsTNMBDInZwbF68uIjuw== 2024-04-25T07:29:20.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQBipmO5+POhAABeqsTNMBDInZwbF68uIjuw=="}' 2024-04-25T07:29:20.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 77b1f0bf-91e8-4654-964f-157876cf2a2c -i td/osd-fast-mark-down/0/new.json 2024-04-25T07:29:21.307 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:29:21.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2024-04-25T07:29:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQBipmO5+POhAABeqsTNMBDInZwbF68uIjuw== --osd-uuid 77b1f0bf-91e8-4654-964f-157876cf2a2c 2024-04-25T07:29:21.366 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:21.366+0000 7f3c309e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:21.384 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:21.386+0000 7f3c309e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:21.386 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:21.386+0000 7f3c309e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:21.386 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:21.390+0000 7f3c309e27c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:29:21.386 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:21.390+0000 7f3c309e27c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:29:21.387 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:21.390+0000 7f3c309e27c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:29:21.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:21.426+0000 7f3c309e27c0 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2024-04-25T07:29:27.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2024-04-25T07:29:27.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:29:27.622 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:29:27.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:29:27.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:29:28.156 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:29:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:29:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:29:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:29:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:29:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:29:28.189 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:28.189+0000 7f219f6437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:28.215 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:28.217+0000 7f219f6437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:28.216 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:28.217+0000 7f219f6437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:28.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:29:28.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:29:28.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:29:28.638 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:29:28.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:29:28.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:29:28.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:28.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:29:28.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:28.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:28.788 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:28.789+0000 7f219f6437c0 -1 Falling back to public interface 2024-04-25T07:29:29.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:30.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:30.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:30.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:29:30.073 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:29:30.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:30.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:30.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:31.397 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:31.397+0000 7f219f6437c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:29:31.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:31.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:31.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:29:31.477 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:29:31.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:31.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:32.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:33.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:33.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:33.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:29:33.021 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:29:33.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:33.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:33.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:34.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:34.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:34.573 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:29:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:29:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:35.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:36.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:36.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:36.017 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:29:36.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:29:36.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:36.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:36.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:37.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:37.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:37.476 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:29:37.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:29:37.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:37.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:37.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:38.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:38.901 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:29:38.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:29:38.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:38.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:39.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:40.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:40.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:40.484 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:29:40.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:29:40.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:40.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:40.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:41.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:41.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:41.907 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:29:41.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:29:41.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:41.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:42.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:43.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:43.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:43.364 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:29:43.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:29:43.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:43.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:43.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:44.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:44.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:44.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:29:44.986 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:29:44.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:44.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:45.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:46.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:46.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:46.473 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:29:46.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:29:46.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:46.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:47.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:48.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:48.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:48.151 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:29:48.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:29:48.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:48.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:49.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:50.245 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:29:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:29:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:51.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:52.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:52.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:52.186 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:29:52.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:29:52.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:52.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:52.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:53.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:53.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:53.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:29:53.780 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:29:53.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:53.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:54.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:55.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:55.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:55.496 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:29:55.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:29:55.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:55.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:56.025 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2007862697,v1:127.0.0.1:6803/2007862697] [v2:127.0.0.1:6804/2007862697,v1:127.0.0.1:6805/2007862697] exists,up 77b1f0bf-91e8-4654-964f-157876cf2a2c 2024-04-25T07:29:56.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:56.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:56.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:56.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2024-04-25T07:29:56.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=162649 2024-04-25T07:29:56.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:29:56.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:56.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:29:56.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:29:56.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:56.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:56.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:56.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:56.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:29:56.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:29:56.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2024-04-25T07:29:56.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:29:56.034 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 d6578a0f-b63b-4ace-bf4b-8e9ca0c32a17 2024-04-25T07:29:56.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6578a0f-b63b-4ace-bf4b-8e9ca0c32a17 2024-04-25T07:29:56.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d6578a0f-b63b-4ace-bf4b-8e9ca0c32a17' 2024-04-25T07:29:56.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:29:56.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0Bipm7KZsAxAAtibDLh89CriZOKDqxDNluQ== 2024-04-25T07:29:56.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0Bipm7KZsAxAAtibDLh89CriZOKDqxDNluQ=="}' 2024-04-25T07:29:56.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6578a0f-b63b-4ace-bf4b-8e9ca0c32a17 -i td/osd-fast-mark-down/1/new.json 2024-04-25T07:29:56.538 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:29:56.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2024-04-25T07:29:56.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0Bipm7KZsAxAAtibDLh89CriZOKDqxDNluQ== --osd-uuid d6578a0f-b63b-4ace-bf4b-8e9ca0c32a17 2024-04-25T07:29:56.586 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:56.585+0000 7f08d0e0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:56.633 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:56.633+0000 7f08d0e0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:56.634 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:56.633+0000 7f08d0e0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:56.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:56.637+0000 7f08d0e0a7c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:29:56.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:56.637+0000 7f08d0e0a7c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:29:56.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:56.637+0000 7f08d0e0a7c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:29:56.742 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:29:56.741+0000 7f08d0e0a7c0 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2024-04-25T07:30:03.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2024-04-25T07:30:03.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:30:03.433 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:30:03.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:30:03.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:30:03.993 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:30:03.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:30:03.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:03.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:30:03.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:30:03.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:30:04.027 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:04.029+0000 7f990ae7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:04.070 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:04.069+0000 7f990ae7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:04.071 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:04.073+0000 7f990ae7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:04.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:30:04.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:04.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:30:04.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:04.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:04.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:04.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:04.456 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:30:04.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:04.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:04.640 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:04.641+0000 7f990ae7a7c0 -1 Falling back to public interface 2024-04-25T07:30:04.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:05.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:05.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:05.901 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:30:05.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:30:05.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:05.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:06.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:07.258 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:07.257+0000 7f990ae7a7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:30:07.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:07.335 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:30:07.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:07.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:30:07.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:07.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:08.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:09.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:09.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:09.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:30:09.052 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:30:09.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:09.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:09.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:10.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:10.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:10.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:30:10.503 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:30:10.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:10.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:10.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:11.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:11.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:11.994 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:30:11.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:30:11.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:11.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:12.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:13.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:13.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:13.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:30:13.470 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:30:13.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:13.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:13.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:14.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:14.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:14.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:30:14.937 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:30:14.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:14.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:15.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:16.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:16.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:16.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:30:16.459 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:30:16.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:16.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:16.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:17.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:17.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:17.915 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:30:17.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:30:17.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:17.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:18.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:19.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:19.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:19.508 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:30:19.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:30:19.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:19.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:19.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:20.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:20.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:20.983 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:30:20.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:30:20.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:20.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:21.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:22.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:22.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:22.934 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:30:22.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:30:22.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:22.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:23.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:24.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:24.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:24.816 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:30:24.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:30:24.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:24.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:25.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:26.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:26.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:26.580 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:30:26.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:30:26.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:26.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:27.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:28.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:28.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:28.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:30:28.386 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:30:28.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:28.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:29.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:30.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:30.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:30.149 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:30:30.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:30:30.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:30.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:30.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:31.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:31.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:31.741 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:30:31.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:30:31.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:31.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:32.259 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1977652965,v1:127.0.0.1:6811/1977652965] [v2:127.0.0.1:6812/1977652965,v1:127.0.0.1:6813/1977652965] exists,up d6578a0f-b63b-4ace-bf4b-8e9ca0c32a17 2024-04-25T07:30:32.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:30:32.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:30:32.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:30:32.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=164390 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2024-04-25T07:30:32.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:32.265 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:30:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2024-04-25T07:30:32.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:30:32.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=04b84b78-cc57-4334-8d62-5d9ad4982478 2024-04-25T07:30:32.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 04b84b78-cc57-4334-8d62-5d9ad4982478' 2024-04-25T07:30:32.268 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 04b84b78-cc57-4334-8d62-5d9ad4982478 2024-04-25T07:30:32.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:30:32.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYBipm1U5gERAAjM+gawSKzVegrfXmDAFpnQ== 2024-04-25T07:30:32.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYBipm1U5gERAAjM+gawSKzVegrfXmDAFpnQ=="}' 2024-04-25T07:30:32.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 04b84b78-cc57-4334-8d62-5d9ad4982478 -i td/osd-fast-mark-down/2/new.json 2024-04-25T07:30:32.859 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:30:32.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2024-04-25T07:30:32.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYBipm1U5gERAAjM+gawSKzVegrfXmDAFpnQ== --osd-uuid 04b84b78-cc57-4334-8d62-5d9ad4982478 2024-04-25T07:30:32.913 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:32.912+0000 7f02a6f5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:32.950 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:32.948+0000 7f02a6f5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:32.952 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:32.952+0000 7f02a6f5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:32.952 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:32.952+0000 7f02a6f5b7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:30:32.952 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:32.952+0000 7f02a6f5b7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:30:32.952 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:32.952+0000 7f02a6f5b7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:30:32.990 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:32.988+0000 7f02a6f5b7c0 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2024-04-25T07:30:39.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2024-04-25T07:30:39.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:30:39.433 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:30:39.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:30:39.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:30:39.999 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:30:39.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:30:39.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:30:40.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:30:40.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ef113ec2-d5ed-487a-b783-9dc52fd6c48c --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.161503/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:40.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:30:40.033 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:40.032+0000 7fcfd2a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:40.035 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:40.036+0000 7fcfd2a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:40.037 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:40.036+0000 7fcfd2a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:40.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:30:40.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:40.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:30:40.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:40.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:40.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:40.392 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:30:40.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:40.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:40.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:40.600 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:40.600+0000 7fcfd2a9c7c0 -1 Falling back to public interface 2024-04-25T07:30:40.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:41.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:41.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:41.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:30:41.838 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:30:41.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:41.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:42.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:43.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:30:43.212+0000 7fcfd2a9c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:30:43.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:43.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:43.251 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:30:43.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:30:43.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:43.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:43.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:44.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:44.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:45.000 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:30:45.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:30:45.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:45.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:45.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:46.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:46.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:46.455 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:30:46.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:30:46.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:46.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:46.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:47.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:47.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:47.901 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:30:47.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:30:47.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:47.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:48.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:49.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:49.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:49.437 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:30:49.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:30:49.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:49.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:49.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:50.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:50.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:50.905 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:30:50.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:30:50.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:50.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:51.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:52.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:52.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:52.459 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:30:52.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:30:52.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:52.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:52.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:53.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:53.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:53.882 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:30:53.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:30:53.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:53.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:54.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:55.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:55.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:55.485 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:30:55.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:30:55.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:55.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:56.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:57.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:57.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:57.018 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:30:57.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:30:57.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:57.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:57.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:58.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:58.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:58.814 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:30:58.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:30:58.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:58.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:59.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:00.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:00.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:00.726 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:31:00.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:31:00.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:00.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:31:01.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:02.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:02.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:02.447 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:31:02.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:31:02.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:02.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:31:03.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:04.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:04.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:04.268 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:31:04.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:31:04.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:04.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:31:05.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:06.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:06.530 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:31:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:31:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:06.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:31:07.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:08.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:08.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:08.422 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:31:08.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:31:08.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:08.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:31:08.854 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3076480297,v1:127.0.0.1:6819/3076480297] [v2:127.0.0.1:6820/3076480297,v1:127.0.0.1:6821/3076480297] exists,up 04b84b78-cc57-4334-8d62-5d9ad4982478 2024-04-25T07:31:08.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:31:08.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:31:08.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:31:08.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2024-04-25T07:31:08.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=166129 2024-04-25T07:31:08.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2024-04-25T07:31:08.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:31:09.476 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-04-25T07:31:09.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:31:09.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:31:10.150 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-04-25T07:31:10.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:31:11.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:31:13.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2024-04-25T07:31:13.846 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T07:31:13.846 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2024-04-25T07:31:13.846 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_167099 2024-04-25T07:31:13.846 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:31:13.846 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:31:14.846 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 65 49 0.191386 0.191406 0.250189 0.272053 2024-04-25T07:31:15.846 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 128 112 0.218721 0.246094 0.343001 0.275995 2024-04-25T07:31:16.846 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 2.23807 2024-04-25T07:31:16.846 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 128 2024-04-25T07:31:16.846 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 4096 2024-04-25T07:31:16.846 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 4096 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.223407 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.0386699 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.246094 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.191406 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 57 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 9.89949 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 63 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 49 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.276681 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0298726 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.343554 2024-04-25T07:31:16.847 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.197544 2024-04-25T07:31:16.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2024-04-25T07:31:17.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2024-04-25T07:31:17.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2024-04-25T07:31:17.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-04-25T07:31:17.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2024-04-25T07:31:17.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=164390 2024-04-25T07:31:17.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164390 -eq 0 ']' 2024-04-25T07:31:17.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=164390 2024-04-25T07:31:17.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 164390 2024-04-25T07:31:17.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-04-25T07:31:17.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-04-25T07:31:17.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-04-25T07:31:17.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-04-25T07:31:18.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-04-25T07:31:18.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-04-25T07:31:18.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-04-25T07:31:18.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-04-25T07:31:18.997 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164390 -eq 164390 ']' 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=162649 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 162649 -eq 164390 ']' 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=162649 2024-04-25T07:31:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 162649 2024-04-25T07:31:19.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-04-25T07:31:19.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-04-25T07:31:19.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-04-25T07:31:19.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-04-25T07:31:20.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-04-25T07:31:20.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-04-25T07:31:20.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-04-25T07:31:20.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-04-25T07:31:20.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-04-25T07:31:20.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2024-04-25T07:31:20.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2024-04-25T07:31:20.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2024-04-25T07:31:20.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-04-25T07:31:20.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2024-04-25T07:31:20.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2024-04-25T07:31:20.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2024-04-25T07:31:20.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:31:20.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:31:20.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:31:20.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:31:20.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:31:20.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:31:20.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:31:20.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:31:21.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:31:21.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:31:21.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:31:21.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:31:21.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:31:21.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:31:21.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:31:21.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:21.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:31:21.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:21.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:31:21.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:31:21.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:31:21.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:31:21.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:31:21.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:31:21.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161503 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-fast-mark-down 0 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:31:21.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:31:21.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:31:21.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:31:21.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:31:21.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:31:21.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:31:21.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:31:21.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:31:21.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:31:21.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:31:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:31:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:31:21.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:31:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:31:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:31:21.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:31:21.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.161503 2024-04-25T07:31:21.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.161503 2024-04-25T07:31:21.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:31:21.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:31:21.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:31:21.080 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:31:21.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:31:21.131 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2024-04-25T07:31:21.131 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2024-04-25T07:31:21.185 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:31:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:31:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:31:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:31:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:31:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:31:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:31:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:31:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-force-create-pg 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none ' 2024-04-25T07:31:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2024-04-25T07:31:21.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2024-04-25T07:31:21.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:31:21.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2024-04-25T07:31:21.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2024-04-25T07:31:21.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2024-04-25T07:31:21.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2024-04-25T07:31:21.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2024-04-25T07:31:21.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:31:21.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:31:21.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:31:21.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:31:21.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:31:21.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:31:21.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:31:21.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:31:21.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:31:21.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:31:21.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:31:21.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:31:21.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:31:21.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:31:21.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:31:21.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:21.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:31:21.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:31:21.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:21.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:31:21.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:31:21.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:31:21.203 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:31:21.203 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.203 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:31:21.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.167230 2024-04-25T07:31:21.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:31:21.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:31:21.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2024-04-25T07:31:21.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:31:21.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:31:21.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.167230 2024-04-25T07:31:21.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:31:21.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2024-04-25T07:31:21.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:31:21.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:31:21.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:31:21.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:31:21.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:31:21.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:31:21.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:31:21.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:31:21.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:31:21.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:31:21.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:31:21.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:31:21.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:31:21.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:31:21.864 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:31:21.864 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:31:21.864 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:31:21.865 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:31:21.865 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.865 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:31:21.865 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.167230/ceph-mon.a.asok 2024-04-25T07:31:21.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:31:21.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.167230/ceph-mon.a.asok config get fsid 2024-04-25T07:31:21.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:31:21.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:31:21.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:31:21.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:31:21.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:31:21.984 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:31:21.984 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:31:21.985 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:31:21.985 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:31:21.985 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.986 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:31:21.986 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.167230/ceph-mon.a.asok 2024-04-25T07:31:21.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:31:21.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.167230/ceph-mon.a.asok config get mon_host 2024-04-25T07:31:22.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2024-04-25T07:31:22.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2024-04-25T07:31:22.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:31:22.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:31:22.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:31:22.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2024-04-25T07:31:22.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:31:23.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:31:23.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:31:23.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:31:23.017 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:31:23.017 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:23.017 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:31:23.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:31:23.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:31:23.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-04-25T07:31:23.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:31:23.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2024-04-25T07:31:23.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:31:23.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9542c85-9c69-4349-93c8-796a57b550b3 2024-04-25T07:31:23.056 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 d9542c85-9c69-4349-93c8-796a57b550b3 2024-04-25T07:31:23.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d9542c85-9c69-4349-93c8-796a57b550b3' 2024-04-25T07:31:23.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:31:23.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLBipmZt9aBBAAnm3BDeHRPnTNRpg0loelXA== 2024-04-25T07:31:23.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLBipmZt9aBBAAnm3BDeHRPnTNRpg0loelXA=="}' 2024-04-25T07:31:23.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9542c85-9c69-4349-93c8-796a57b550b3 -i td/osd-force-create-pg/0/new.json 2024-04-25T07:31:23.373 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:31:23.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2024-04-25T07:31:23.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLBipmZt9aBBAAnm3BDeHRPnTNRpg0loelXA== --osd-uuid d9542c85-9c69-4349-93c8-796a57b550b3 2024-04-25T07:31:23.431 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:23.431+0000 7f2dca2bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:23.433 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:23.431+0000 7f2dca2bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:23.434 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:23.435+0000 7f2dca2bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:23.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:23.435+0000 7f2dca2bf7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:31:23.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:23.435+0000 7f2dca2bf7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:31:23.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:23.435+0000 7f2dca2bf7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:31:23.473 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:23.471+0000 7f2dca2bf7c0 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2024-04-25T07:31:29.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2024-04-25T07:31:29.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:31:29.626 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:31:29.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:31:29.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:31:30.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:31:30.209 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:31:30.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:31:30.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:31:30.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:31:30.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:31:30.238 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:30.239+0000 7fd6ac6037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:30.254 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:30.255+0000 7fd6ac6037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:30.255 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:30.255+0000 7fd6ac6037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:30.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:31:30.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:31:30.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:31:30.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:31:30.660 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:31:30.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:31:30.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:30.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:31:30.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:30.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:30.852 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:30.851+0000 7fd6ac6037c0 -1 Falling back to public interface 2024-04-25T07:31:31.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:32.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:32.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:32.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:31:32.109 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:31:32.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:32.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:32.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:33.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:33.419+0000 7fd6ac6037c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:31:33.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:33.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:33.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:31:33.508 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:31:33.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:33.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:34.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:35.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:35.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:35.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:31:35.135 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:31:35.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:35.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:35.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:36.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:36.578 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:31:36.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:36.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:31:36.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:36.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:37.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:38.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:38.007 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:31:38.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:38.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:31:38.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:38.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:38.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:39.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:39.530 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:31:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:31:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:40.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:41.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:41.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:41.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:31:41.004 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:31:41.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:41.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:41.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:42.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:42.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:42.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:31:42.593 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:31:42.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:42.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:43.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:44.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:44.019 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:31:44.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:44.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:31:44.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:44.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:44.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:45.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:45.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:45.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:31:45.432 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:31:45.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:45.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:45.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:46.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:46.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:46.839 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:31:46.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:31:46.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:46.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:47.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:48.640 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:31:48.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:48.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:48.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:31:48.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:48.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:50.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:50.606 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:31:50.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:50.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:31:50.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:50.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:51.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:52.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:52.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:52.514 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:31:52.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:31:52.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:52.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:53.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:54.232 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:31:54.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:54.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:54.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:31:54.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:54.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:55.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:56.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:56.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:56.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:31:56.470 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:31:56.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:56.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:57.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:58.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:58.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:58.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:31:58.078 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:31:58.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:58.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:58.533 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4023662246,v1:127.0.0.1:6803/4023662246] [v2:127.0.0.1:6804/4023662246,v1:127.0.0.1:6805/4023662246] exists,up d9542c85-9c69-4349-93c8-796a57b550b3 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:31:58.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:31:58.538 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:31:58.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:31:58.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:31:58.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2024-04-25T07:31:58.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:31:58.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc557746-85c1-43c2-a3bd-e4f001d03e13 2024-04-25T07:31:58.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bc557746-85c1-43c2-a3bd-e4f001d03e13' 2024-04-25T07:31:58.540 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 bc557746-85c1-43c2-a3bd-e4f001d03e13 2024-04-25T07:31:58.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:31:58.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuBipm+8ZiIRAAEdyAuzBsRaDx6Ssc7O8Pgg== 2024-04-25T07:31:58.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuBipm+8ZiIRAAEdyAuzBsRaDx6Ssc7O8Pgg=="}' 2024-04-25T07:31:58.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc557746-85c1-43c2-a3bd-e4f001d03e13 -i td/osd-force-create-pg/1/new.json 2024-04-25T07:31:59.028 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:31:59.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2024-04-25T07:31:59.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuBipm+8ZiIRAAEdyAuzBsRaDx6Ssc7O8Pgg== --osd-uuid bc557746-85c1-43c2-a3bd-e4f001d03e13 2024-04-25T07:31:59.076 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:59.075+0000 7f88bf5937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:59.106 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:59.107+0000 7f88bf5937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:59.108 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:59.107+0000 7f88bf5937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:59.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:59.107+0000 7f88bf5937c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:31:59.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:59.107+0000 7f88bf5937c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:31:59.109 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:59.107+0000 7f88bf5937c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:31:59.145 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:31:59.143+0000 7f88bf5937c0 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2024-04-25T07:32:05.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2024-04-25T07:32:05.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:32:05.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:32:05.667 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:32:05.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:32:06.224 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:32:06.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:32:06.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:32:06.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:32:06.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:32:06.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:32:06.253 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:06.250+0000 7f16597ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:06.283 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:06.282+0000 7f16597ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:06.285 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:06.282+0000 7f16597ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:06.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:32:06.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:32:06.652 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:32:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:32:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:32:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:32:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:32:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:06.848 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:06.846+0000 7f16597ad7c0 -1 Falling back to public interface 2024-04-25T07:32:07.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:08.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:08.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:08.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:32:08.075 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:32:08.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:08.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:08.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:09.382 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:09.378+0000 7f16597ad7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:32:09.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:09.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:09.513 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:32:09.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:32:09.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:09.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:10.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:32:11.120 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:32:11.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:11.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:11.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:12.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:12.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:12.556 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:32:12.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:32:12.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:12.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:13.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:14.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:14.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:14.027 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:32:14.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:32:14.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:14.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:14.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:15.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:15.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:15.685 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:32:15.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:32:15.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:15.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:16.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:17.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:17.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:17.268 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:32:17.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:32:17.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:17.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:17.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:18.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:18.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:18.680 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:32:18.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:32:18.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:18.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:19.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:20.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:20.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:20.249 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:32:20.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:32:20.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:20.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:20.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:21.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:21.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:21.769 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:32:21.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:32:21.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:21.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:22.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:23.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:23.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:23.210 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:32:23.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:32:23.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:23.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:24.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:25.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:25.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:25.032 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:32:25.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:32:25.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:25.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:25.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:26.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:26.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:26.681 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:32:26.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:32:26.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:26.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:27.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:28.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:28.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:28.812 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:32:28.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:32:28.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:28.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:29.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:30.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:30.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:30.725 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:32:30.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:32:30.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:30.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:31.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:32.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:32.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:32.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:32:32.529 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:32:32.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:32.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:33.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:34.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:34.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:34.318 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:32:34.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:32:34.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:34.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:34.739 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/934162158,v1:127.0.0.1:6811/934162158] [v2:127.0.0.1:6812/934162158,v1:127.0.0.1:6813/934162158] exists,up bc557746-85c1-43c2-a3bd-e4f001d03e13 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:32:34.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:32:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:32:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2024-04-25T07:32:34.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:32:34.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6252fb84-193b-4b0c-8a79-ef266e19691f 2024-04-25T07:32:34.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6252fb84-193b-4b0c-8a79-ef266e19691f' 2024-04-25T07:32:34.745 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 6252fb84-193b-4b0c-8a79-ef266e19691f 2024-04-25T07:32:34.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:32:34.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASBypmJk3BLRAAhoJfn1QqyiHLvexqSq2KJw== 2024-04-25T07:32:34.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASBypmJk3BLRAAhoJfn1QqyiHLvexqSq2KJw=="}' 2024-04-25T07:32:34.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6252fb84-193b-4b0c-8a79-ef266e19691f -i td/osd-force-create-pg/2/new.json 2024-04-25T07:32:35.275 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:32:35.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2024-04-25T07:32:35.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASBypmJk3BLRAAhoJfn1QqyiHLvexqSq2KJw== --osd-uuid 6252fb84-193b-4b0c-8a79-ef266e19691f 2024-04-25T07:32:35.330 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:35.330+0000 7f6901f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:35.350 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:35.346+0000 7f6901f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:35.350+0000 7f6901f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:35.350+0000 7f6901f317c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:32:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:35.350+0000 7f6901f317c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:32:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:35.350+0000 7f6901f317c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:32:35.389 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:35.386+0000 7f6901f317c0 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2024-04-25T07:32:41.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2024-04-25T07:32:41.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:32:41.969 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:32:41.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:32:41.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:32:42.522 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:32:42.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:32:42.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:32:42.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:32:42.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:32:42.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:32:42.551 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:42.554+0000 7f92102667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:42.554 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:42.554+0000 7f92102667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:42.555 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:42.558+0000 7f92102667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:42.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:32:42.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:32:42.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:32:42.985 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:32:42.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:32:42.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:32:42.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:42.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:32:42.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:42.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:43.120 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:43.122+0000 7f92102667c0 -1 Falling back to public interface 2024-04-25T07:32:43.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:44.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:44.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:44.406 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:32:44.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:32:44.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:44.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:44.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:45.832 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:32:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:32:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:45.888 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:32:45.890+0000 7f92102667c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:32:46.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:47.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:47.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:47.453 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:32:47.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:32:47.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:47.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:47.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:48.990 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:32:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:32:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:48.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:49.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:50.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:50.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:50.434 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:32:50.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:32:50.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:50.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:50.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:51.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:51.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:51.949 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:32:51.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:32:51.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:51.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:52.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:53.437 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:32:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:32:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:53.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:54.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:54.975 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:32:54.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:54.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:32:54.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:54.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:55.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:56.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:56.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:56.432 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:32:56.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:32:56.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:56.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:56.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:57.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:57.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:57.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:32:57.946 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:32:57.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:57.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:32:58.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:59.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:59.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:59.450 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:32:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:32:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:00.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:01.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:01.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:01.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:33:01.315 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:33:01.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:01.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:02.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:03.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:03.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:03.193 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:33:03.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:33:03.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:03.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:04.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:05.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:05.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:05.079 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:33:05.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:33:05.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:05.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:05.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:06.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:06.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:06.915 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:33:06.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:33:06.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:06.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:07.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:08.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:08.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:08.538 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:33:08.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:33:08.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:08.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:09.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:10.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:10.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:10.255 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:33:10.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:33:10.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:10.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:10.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:11.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:11.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:11.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:33:11.744 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T07:33:11.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:11.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:12.164 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3520371639,v1:127.0.0.1:6819/3520371639] [v2:127.0.0.1:6820/3520371639,v1:127.0.0.1:6821/3520371639] exists,up 6252fb84-193b-4b0c-8a79-ef266e19691f 2024-04-25T07:33:12.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:12.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:12.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:12.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2024-04-25T07:33:12.696 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T07:33:12.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2024-04-25T07:33:12.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:33:12.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:33:12.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:33:12.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:33:12.715 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:33:12.715 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:33:12.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:33:12.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:33:12.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:33:12.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:33:12.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:33:12.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:33:12.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:33:12.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:33:12.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:33:13.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:33:13.540 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:33:13.541 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:33:13.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:33:13.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:13.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:33:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-04-25T07:33:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-04-25T07:33:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-04-25T07:33:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:13.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:33:14.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-04-25T07:33:14.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-04-25T07:33:14.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672977' 2024-04-25T07:33:14.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:14.157 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:33:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-04-25T07:33:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-04-25T07:33:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672977 2-64424509451' 2024-04-25T07:33:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:14.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-04-25T07:33:14.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:14.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:33:14.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-04-25T07:33:14.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:14.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-04-25T07:33:14.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-25T07:33:14.358 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836503 2024-04-25T07:33:14.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:33:14.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836503 2024-04-25T07:33:14.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:33:15.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:33:15.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:33:16.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836503 2024-04-25T07:33:16.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:16.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-04-25T07:33:16.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:16.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:33:16.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-04-25T07:33:16.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:16.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-04-25T07:33:16.431 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672977 2024-04-25T07:33:16.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-25T07:33:16.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:33:16.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672977 2024-04-25T07:33:16.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:16.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-04-25T07:33:16.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:16.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:33:16.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-04-25T07:33:16.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:16.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-04-25T07:33:16.883 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509451 2024-04-25T07:33:16.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-25T07:33:16.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:33:17.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-25T07:33:17.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:33:17.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:33:17.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:33:17.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-04-25T07:33:17.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:33:17.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:33:17.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:33:17.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:33:17.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:33:17.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:33:17.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:33:18.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=36 2024-04-25T07:33:18.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:33:18.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:33:18.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:33:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 36 = 50 2024-04-25T07:33:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 36 '!=' -1 2024-04-25T07:33:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:33:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=36 2024-04-25T07:33:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:33:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:33:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:33:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:33:18.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:33:18.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:33:18.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:33:18.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:33:18.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:33:18.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:33:19.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=36 2024-04-25T07:33:19.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:33:19.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:33:19.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:33:19.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 36 = 50 2024-04-25T07:33:19.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 36 '!=' 36 2024-04-25T07:33:19.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:33:19.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:33:19.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:33:19.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:33:19.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:33:19.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:33:19.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:33:20.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:33:20.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:33:20.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T07:33:20.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:33:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:33:20.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:33:20.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:33:20.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:33:20.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:33:20.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:33:20.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:33:20.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:33:20.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:33:20.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-04-25T07:33:20.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:33:20.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:33:20.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:33:21.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-04-25T07:33:21.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:33:21.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:33:21.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2024-04-25T07:33:21.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:33:21.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:33:21.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:33:21.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:33:21.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:33:21.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:33:21.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2024-04-25T07:33:21.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:33:21.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:33:21.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:33:21.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:33:21.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:33:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:33:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2024-04-25T07:33:22.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:33:22.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:33:22.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:33:22.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:33:22.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:33:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:33:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2024-04-25T07:33:23.644 INFO:tasks.workunit.client.0.smithi088.stderr:PG '1.0' not found 2024-04-25T07:33:23.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2024-04-25T07:33:25.149 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T07:33:25.149 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T07:33:25.149 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T07:33:25.272 INFO:tasks.workunit.client.0.smithi088.stdout:Remove successful 2024-04-25T07:33:25.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2024-04-25T07:33:27.044 INFO:tasks.workunit.client.0.smithi088.stderr:PG '1.0' not found 2024-04-25T07:33:27.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2024-04-25T07:33:27.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:33:27.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:33:27.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/0 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:33:27.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:33:27.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:27.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:27.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:27.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:27.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:33:27.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/0 2024-04-25T07:33:27.594 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:33:27.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:33:27.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:27.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/0/whoami 2024-04-25T07:33:27.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:33:27.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:33:27.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:33:27.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:33:27.625 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:27.625+0000 7f80667bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:27.628 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:27.629+0000 7f80667bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:27.629 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:27.629+0000 7f80667bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:28.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:33:28.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:28.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:33:28.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:28.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:28.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:28.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:28.027 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:33:28.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:28.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:33:28.167 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:28.169+0000 7f80667bf7c0 -1 Falling back to public interface 2024-04-25T07:33:28.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:29.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:29.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:29.446 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:33:29.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:29.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:29.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:33:29.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:30.825 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:30.825+0000 7f80667bf7c0 -1 osd.0 23 log_to_monitors true 2024-04-25T07:33:30.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:30.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:30.884 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:33:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:33:31.401 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 28 up_thru 18 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/2654475426,v1:127.0.0.1:6803/2654475426] [v2:127.0.0.1:6804/2654475426,v1:127.0.0.1:6805/2654475426] exists,up d9542c85-9c69-4349-93c8-796a57b550b3 2024-04-25T07:33:31.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:31.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/1 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:31.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:33:31.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:33:31.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:33:31.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:31.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:31.405 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:31.405 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:31.405 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:33:31.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:33:31.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:33:31.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:31.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:33:31.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:33:31.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:31.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:31.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:31.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:31.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:33:31.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:33:31.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/1 2024-04-25T07:33:31.407 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:33:31.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:33:31.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:31.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/1/whoami 2024-04-25T07:33:31.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:33:31.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:33:31.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:33:31.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:33:31.434 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:31.433+0000 7f723fc8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:31.521 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:31.521+0000 7f723fc8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:31.522 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:31.521+0000 7f723fc8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:31.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:33:31.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:31.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:33:31.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:31.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:31.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:31.933 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:33:31.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:31.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:31.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:32.112 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:32.113+0000 7f723fc8a7c0 -1 Falling back to public interface 2024-04-25T07:33:32.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:33.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:33.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:33.380 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:33:33.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:33.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:33.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:33.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:34.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:34.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:34.800 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:33:34.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:34.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:34.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:34.856 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:34.856+0000 7f723fc8a7c0 -1 osd.1 24 log_to_monitors true 2024-04-25T07:33:35.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:36.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:36.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:36.230 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:33:36.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:33:36.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:36.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:36.647 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 25 last_clean_interval [10,24) [v2:127.0.0.1:6810/3044179272,v1:127.0.0.1:6811/3044179272] [v2:127.0.0.1:6812/3044179272,v1:127.0.0.1:6813/3044179272] exists,up bc557746-85c1-43c2-a3bd-e4f001d03e13 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/2 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:36.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:33:36.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:33:36.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/2 2024-04-25T07:33:36.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:33:36.654 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:33:36.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5fba6728-c805-453d-987c-2bcfcb4c422f --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.167230/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:36.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/2/whoami 2024-04-25T07:33:36.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:33:36.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:33:36.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:33:36.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:33:36.682 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:36.680+0000 7f8ad61667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:36.684 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:36.684+0000 7f8ad61667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:36.684+0000 7f8ad61667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:37.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:33:37.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:37.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:33:37.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:37.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:37.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:37.056 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:33:37.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:37.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:37.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:37.240 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:37.240+0000 7f8ad61667c0 -1 Falling back to public interface 2024-04-25T07:33:37.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:38.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:38.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:38.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:38.483 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:33:38.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:38.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:38.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:39.791 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:39.792+0000 7f8ad61667c0 -1 osd.2 25 log_to_monitors true 2024-04-25T07:33:39.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:39.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:39.911 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:33:39.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:39.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:39.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:40.498 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 34 up_thru 17 down_at 26 last_clean_interval [15,25) [v2:127.0.0.1:6818/1901838606,v1:127.0.0.1:6819/1901838606] [v2:127.0.0.1:6820/1901838606,v1:127.0.0.1:6821/1901838606] exists,up 6252fb84-193b-4b0c-8a79-ef266e19691f 2024-04-25T07:33:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:40.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:40.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2024-04-25T07:33:50.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2024-04-25T07:33:50.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2024-04-25T07:33:50.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2024-04-25T07:33:50.917 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 stale+active+clean 37s 0'0 23:18 [1]p1 [1]p1 2024-04-25T07:33:12.520223+0000 2024-04-25T07:33:12.520223+0000 0 periodic scrub scheduled @ 2024-04-26T17:50:53.185244+0000 2024-04-25T07:33:50.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2024-04-25T07:33:52.048 INFO:tasks.workunit.client.0.smithi088.stderr:pg 1.0 already creating 2024-04-25T07:33:52.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2024-04-25T07:33:52.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:33:52.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:33:52.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:33:52.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:33:52.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:33:52.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:33:52.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:33:52.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:33:52.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:33:52.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:33:52.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:33:52.590 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:33:52.590 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T07:33:52.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:33:52.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:52.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:33:52.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=120259084295 2024-04-25T07:33:52.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 120259084295 2024-04-25T07:33:52.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-120259084295' 2024-04-25T07:33:52.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:52.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:33:52.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=133143986182 2024-04-25T07:33:52.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 133143986182 2024-04-25T07:33:52.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-120259084295 1-133143986182' 2024-04-25T07:33:52.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:33:52.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:33:53.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888069 2024-04-25T07:33:53.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888069 2024-04-25T07:33:53.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-120259084295 1-133143986182 2-146028888069' 2024-04-25T07:33:53.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:53.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-120259084295 2024-04-25T07:33:53.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:53.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:33:53.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-120259084295 2024-04-25T07:33:53.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:53.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=120259084295 2024-04-25T07:33:53.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 120259084295' 2024-04-25T07:33:53.055 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 120259084295 2024-04-25T07:33:53.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:33:53.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084293 -lt 120259084295 2024-04-25T07:33:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:33:54.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:33:54.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:33:54.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084295 -lt 120259084295 2024-04-25T07:33:54.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:54.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:54.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-133143986182 2024-04-25T07:33:54.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:33:54.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-133143986182 2024-04-25T07:33:54.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:54.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=133143986182 2024-04-25T07:33:54.851 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 133143986182 2024-04-25T07:33:54.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 133143986182' 2024-04-25T07:33:54.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:33:55.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 133143986182 -lt 133143986182 2024-04-25T07:33:55.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:33:55.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-146028888069 2024-04-25T07:33:55.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:33:55.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:33:55.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-146028888069 2024-04-25T07:33:55.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:33:55.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888069 2024-04-25T07:33:55.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 146028888069' 2024-04-25T07:33:55.241 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 146028888069 2024-04-25T07:33:55.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:33:55.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888069 -lt 146028888069 2024-04-25T07:33:55.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:33:55.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:33:55.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:33:56.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-04-25T07:33:56.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:33:56.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:33:56.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:33:56.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:33:56.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:33:56.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:33:56.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:33:56.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-04-25T07:33:56.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:33:56.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:33:56.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:33:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-04-25T07:33:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:33:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:33:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2024-04-25T07:33:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:33:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:33:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:33:57.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:33:57.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:33:57.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:33:57.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:33:57.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:33:57.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:33:57.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:33:57.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:33:57.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:33:57.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:33:57.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:33:57.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:33:57.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:33:57.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:33:57.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:33:57.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:33:57.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:33:57.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:33:57.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:33:57.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:33:57.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:57.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:33:57.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.167230 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-force-create-pg 0 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:33:57.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:33:57.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:33:57.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:33:57.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:33:57.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:33:57.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:33:57.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:33:57.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:33:57.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:33:57.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:33:57.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:33:57.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:33:57.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:33:57.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:33:57.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:33:57.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:33:57.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:33:57.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:33:57.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:57.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.167230 2024-04-25T07:33:57.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.167230 2024-04-25T07:33:57.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:33:57.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:33:57.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:33:57.464 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:33:57.464 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:33:57.518 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2024-04-25T07:33:57.518 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2024-04-25T07:33:57.575 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:33:57.579 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:33:57.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-markdown 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2024-04-25T07:33:57.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2024-04-25T07:33:57.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2024-04-25T07:33:57.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2024-04-25T07:33:57.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none ' 2024-04-25T07:33:57.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2024-04-25T07:33:57.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2024-04-25T07:33:57.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_markdown_boot_exceed_time 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_markdown_exceed_maxdown_count 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_osd_stop' 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:33:57.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:33:57.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:33:57.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:33:57.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:33:57.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:33:57.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:33:57.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:33:57.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:33:57.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:33:57.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:33:57.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:33:57.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:33:57.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:33:57.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:33:57.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:33:57.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:33:57.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:33:57.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:33:57.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:33:57.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:57.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:33:57.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:33:57.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:33:57.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:33:57.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:33:57.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:33:57.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:57.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:33:57.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176472 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:33:57.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:33:57.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:33:58.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:33:58.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:58.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:58.074 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:58.074 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:58.074 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:33:58.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:33:58.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:33:58.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:33:58.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:33:58.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:33:58.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:33:58.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:33:58.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:33:58.287 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:33:58.287 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:33:58.287 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:33:58.287 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:33:58.287 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:58.287 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:33:58.287 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176472/ceph-mon.a.asok 2024-04-25T07:33:58.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:33:58.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176472/ceph-mon.a.asok config get fsid 2024-04-25T07:33:58.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:33:58.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:33:58.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:33:58.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:33:58.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:33:58.401 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:33:58.401 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:33:58.401 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:33:58.401 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:33:58.401 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:58.401 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:33:58.401 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176472/ceph-mon.a.asok 2024-04-25T07:33:58.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:33:58.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176472/ceph-mon.a.asok config get mon_host 2024-04-25T07:33:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2024-04-25T07:33:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:33:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:33:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:33:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:33:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:33:58.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:33:59.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:33:59.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:59.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:59.389 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:59.389 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:59.389 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:33:59.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:33:59.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:33:59.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:33:59.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2024-04-25T07:33:59.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:33:59.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:33:59.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:33:59.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:33:59.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:33:59.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:59.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:33:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:33:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:33:59.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:33:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1e09cb40-8e47-410d-8a54-7e74759cd92b 2024-04-25T07:33:59.430 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 1e09cb40-8e47-410d-8a54-7e74759cd92b 2024-04-25T07:33:59.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1e09cb40-8e47-410d-8a54-7e74759cd92b' 2024-04-25T07:33:59.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:33:59.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnBypmWlGlGhAAsWm72tnXAaqUjKJpNiEy/g== 2024-04-25T07:33:59.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnBypmWlGlGhAAsWm72tnXAaqUjKJpNiEy/g=="}' 2024-04-25T07:33:59.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1e09cb40-8e47-410d-8a54-7e74759cd92b -i td/osd-markdown/0/new.json 2024-04-25T07:33:59.720 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:33:59.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:33:59.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnBypmWlGlGhAAsWm72tnXAaqUjKJpNiEy/g== --osd-uuid 1e09cb40-8e47-410d-8a54-7e74759cd92b 2024-04-25T07:33:59.763 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:59.764+0000 7fedc66f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:59.789 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:59.788+0000 7fedc66f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:59.791 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:59.792+0000 7fedc66f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:59.791 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:59.792+0000 7fedc66f97c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:33:59.792 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:59.792+0000 7fedc66f97c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:33:59.792 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:59.792+0000 7fedc66f97c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:33:59.829 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:33:59.828+0000 7fedc66f97c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:34:05.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:34:05.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:34:05.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:34:05.982 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:34:05.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:34:06.521 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:34:06.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:34:06.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:34:06.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:34:06.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:34:06.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:34:06.550 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:06.548+0000 7f09fe9727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:06.552 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:06.552+0000 7f09fe9727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:06.554 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:06.552+0000 7f09fe9727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:06.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:34:06.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:34:06.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:34:06.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:34:06.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:34:06.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:06.925 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:34:06.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:34:06.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:06.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:07.112 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:07.112+0000 7f09fe9727c0 -1 Falling back to public interface 2024-04-25T07:34:07.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:08.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:08.319 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:34:08.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:08.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:34:08.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:08.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:08.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:09.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:09.636+0000 7f09fe9727c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:34:09.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:09.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:09.719 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:34:09.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:34:09.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:09.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:10.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:11.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:11.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:11.347 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:34:11.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:34:11.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:11.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:11.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:12.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:12.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:12.849 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:34:12.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:34:12.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:12.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:13.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:14.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:14.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:14.285 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:34:14.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:34:14.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:14.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:14.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:15.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:15.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:15.803 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:34:15.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:34:15.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:15.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:16.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:17.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:17.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:17.259 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:34:17.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:34:17.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:17.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:17.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:18.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:18.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:18.856 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:34:18.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:34:18.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:18.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:19.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:20.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:20.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:20.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:34:20.304 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:34:20.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:20.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:20.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:21.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:21.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:21.874 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:34:21.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:34:21.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:21.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:22.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:23.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:23.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:23.298 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:34:23.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:34:23.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:23.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:24.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:25.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:25.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:25.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:34:25.049 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:34:25.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:25.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:25.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:26.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:26.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:26.824 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:34:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:34:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:27.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:28.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:28.661 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:34:28.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:28.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:34:28.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:28.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:30.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:30.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:30.656 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:34:30.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:34:30.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:30.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:31.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:32.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:32.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:32.476 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:34:32.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:34:32.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:32.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:33.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:34.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:34.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:34.550 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:34:34.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:34:34.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:34.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:34:34.987 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2552957838,v1:127.0.0.1:6803/2552957838] [v2:127.0.0.1:6804/2552957838,v1:127.0.0.1:6805/2552957838] exists,up 1e09cb40-8e47-410d-8a54-7e74759cd92b 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:34:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:34:34.992 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:34:34.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:34:34.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7eb2d810-8dc5-4123-9b26-480da26f575c 2024-04-25T07:34:34.994 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 7eb2d810-8dc5-4123-9b26-480da26f575c 2024-04-25T07:34:34.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7eb2d810-8dc5-4123-9b26-480da26f575c' 2024-04-25T07:34:34.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:34:35.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLBypmPd7MABAAwkxg5JDNO5dUd6WyckR46w== 2024-04-25T07:34:35.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLBypmPd7MABAAwkxg5JDNO5dUd6WyckR46w=="}' 2024-04-25T07:34:35.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7eb2d810-8dc5-4123-9b26-480da26f575c -i td/osd-markdown/1/new.json 2024-04-25T07:34:35.495 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:34:35.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:34:35.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLBypmPd7MABAAwkxg5JDNO5dUd6WyckR46w== --osd-uuid 7eb2d810-8dc5-4123-9b26-480da26f575c 2024-04-25T07:34:35.537 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:35.535+0000 7f99969de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:35.572 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:35.571+0000 7f99969de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:35.573 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:35.571+0000 7f99969de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:35.574 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:35.571+0000 7f99969de7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:34:35.574 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:35.571+0000 7f99969de7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:34:35.574 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:35.571+0000 7f99969de7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:34:35.612 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:35.611+0000 7f99969de7c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:34:42.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:34:42.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:34:42.289 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:34:42.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:34:42.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:34:42.838 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:34:42.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:34:42.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:34:42.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:34:42.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:34:42.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:34:42.867 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:42.867+0000 7f7e0cc107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:42.902 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:42.899+0000 7f7e0cc107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:42.903 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:42.903+0000 7f7e0cc107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:43.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:34:43.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:34:43.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:34:43.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:34:43.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:34:43.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:43.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:34:43.280 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:34:43.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:43.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:43.473 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:43.471+0000 7f7e0cc107c0 -1 Falling back to public interface 2024-04-25T07:34:43.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:44.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:44.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:44.705 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:34:44.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:34:44.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:44.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:45.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:46.098 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:34:46.095+0000 7f7e0cc107c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:34:46.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:46.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:46.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:34:46.107 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:34:46.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:46.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:46.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:47.502 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:34:47.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:47.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:47.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:34:47.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:47.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:48.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:49.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:49.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:49.023 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:34:49.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:34:49.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:49.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:49.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:50.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:50.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:50.479 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:34:50.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:34:50.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:50.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:50.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:51.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:51.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:51.952 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:34:51.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:34:51.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:51.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:52.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:53.395 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:34:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:34:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:53.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:54.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:54.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:54.799 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:34:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:34:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:55.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:56.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:56.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:56.276 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:34:56.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:34:56.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:56.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:56.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:57.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:57.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:57.784 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:34:57.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:34:57.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:57.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:34:58.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:59.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:59.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:59.381 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:34:59.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:34:59.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:59.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:35:00.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:01.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:01.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:01.078 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:35:01.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:35:01.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:01.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:35:01.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:02.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:02.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:02.807 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:35:02.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:35:02.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:02.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:35:04.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:05.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:05.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:05.061 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:35:05.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:35:05.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:05.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:35:06.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:07.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:07.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:07.050 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:35:07.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:35:07.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:07.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:35:07.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:08.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:08.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:08.817 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:35:08.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:35:08.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:08.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:35:09.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:10.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:10.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:10.652 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:35:10.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:35:10.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:10.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:35:11.248 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3685036769,v1:127.0.0.1:6811/3685036769] [v2:127.0.0.1:6812/3685036769,v1:127.0.0.1:6813/3685036769] exists,up 7eb2d810-8dc5-4123-9b26-480da26f575c 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:35:11.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:35:11.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:35:11.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:35:11.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:35:11.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:35:11.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:35:11.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:35:11.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:35:11.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:35:11.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:35:11.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=174e661a-020b-4fec-9914-e8066595ef1c 2024-04-25T07:35:11.256 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 174e661a-020b-4fec-9914-e8066595ef1c 2024-04-25T07:35:11.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 174e661a-020b-4fec-9914-e8066595ef1c' 2024-04-25T07:35:11.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:35:11.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvBypm2gSTEBAAZseRsnMagVPPD1fxKJpmUw== 2024-04-25T07:35:11.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvBypm2gSTEBAAZseRsnMagVPPD1fxKJpmUw=="}' 2024-04-25T07:35:11.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 174e661a-020b-4fec-9914-e8066595ef1c -i td/osd-markdown/2/new.json 2024-04-25T07:35:11.805 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:35:11.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:35:11.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvBypm2gSTEBAAZseRsnMagVPPD1fxKJpmUw== --osd-uuid 174e661a-020b-4fec-9914-e8066595ef1c 2024-04-25T07:35:11.852 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:11.850+0000 7fc41dbdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:11.881 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:11.878+0000 7fc41dbdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:11.883 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:11.882+0000 7fc41dbdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:11.883 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:11.882+0000 7fc41dbdd7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:35:11.883 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:11.882+0000 7fc41dbdd7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:35:11.883 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:11.882+0000 7fc41dbdd7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:35:11.921 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:11.918+0000 7fc41dbdd7c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:35:18.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:35:18.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:35:18.849 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:35:18.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:35:18.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:35:19.404 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:35:19.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:35:19.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:35:19.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:35:19.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:35:19.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:35:19.434 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:19.430+0000 7fad7b5f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:19.461 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:19.458+0000 7fad7b5f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:19.462 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:19.458+0000 7fad7b5f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:35:19.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:35:19.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:35:19.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:35:19.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:35:19.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:35:19.835 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:35:19.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:19.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:35:19.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:19.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:20.044 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:20.042+0000 7fad7b5f27c0 -1 Falling back to public interface 2024-04-25T07:35:20.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:35:21.325 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:35:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:21.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:21.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:22.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:22.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:22.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:35:22.747 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:35:22.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:22.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:22.753 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:35:22.750+0000 7fad7b5f27c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:35:23.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:24.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:24.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:24.145 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:35:24.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:35:24.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:24.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:24.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:25.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:25.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:25.679 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:35:25.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:35:25.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:25.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:26.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:27.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:27.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:27.197 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:35:27.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:35:27.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:27.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:27.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:28.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:28.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:28.700 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:35:28.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:35:28.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:28.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:29.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:30.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:30.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:30.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:35:30.177 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:35:30.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:30.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:30.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:31.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:31.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:31.664 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:35:31.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:35:31.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:31.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:32.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:33.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:33.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:33.070 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:35:33.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:35:33.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:33.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:33.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:34.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:34.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:34.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:35:34.712 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:35:34.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:34.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:35.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:36.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:36.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:36.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:35:36.218 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:35:36.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:36.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:36.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:37.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:37.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:37.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:35:37.763 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:35:37.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:37.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:38.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:39.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:39.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:39.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:35:39.299 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:35:39.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:39.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:40.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:41.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:41.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:41.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:35:41.091 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:35:41.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:41.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:41.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:42.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:42.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:42.918 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:35:42.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:35:42.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:42.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:43.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:35:44.837 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:35:44.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:44.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:45.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:46.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:46.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:46.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:35:46.756 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:35:46.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:46.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:47.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:35:48.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:35:48.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:35:48.613 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T07:35:48.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:35:48.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:35:48.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:35:49.047 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/790347532,v1:127.0.0.1:6819/790347532] [v2:127.0.0.1:6820/790347532,v1:127.0.0.1:6821/790347532] exists,up 174e661a-020b-4fec-9914-e8066595ef1c 2024-04-25T07:35:49.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:35:49.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:35:49.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:35:49.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2024-04-25T07:35:49.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:35:49.457 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-04-25T07:35:49.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:35:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:35:50.031 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-04-25T07:35:50.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:35:51.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:35:53.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2024-04-25T07:35:53.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2024-04-25T07:35:53.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2024-04-25T07:35:53.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:35:53.240 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:35:53.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2024-04-25T07:35:53.388 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:35:53.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2024-04-25T07:35:53.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2024-04-25T07:35:53.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2024-04-25T07:35:53.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:35:53.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2024-04-25T07:35:53.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:35:53.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:35:53.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:35:53.948 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:35:53.948 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:35:53.948 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:35:53.948 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:35:53.948 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:35:53.948 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:35:53.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:35:53.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:35:53.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:35:54.362 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:35:54.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:35:54.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:35:54.880 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:35:54.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:36:04.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:36:04.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:36:05.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:36:05.448 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:36:05.448 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:36:05.448 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:36:05.448 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:36:05.448 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:36:05.449 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:36:05.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:36:05.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:36:05.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:36:05.935 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:36:05.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:36:05.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:36:06.349 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:36:06.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:36:16.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:36:16.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:36:16.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:36:16.947 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:36:16.947 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:36:16.947 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:36:16.947 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:36:16.947 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:36:16.947 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:36:16.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:36:16.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:36:16.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:36:17.380 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:36:17.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:36:17.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:36:17.780 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:36:17.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:36:27.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2024-04-25T07:36:42.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:36:42.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2024-04-25T07:36:42.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2024-04-25T07:36:42.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2024-04-25T07:36:43.351 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:36:43.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:36:43.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:36:43.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:36:43.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:36:43.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:36:43.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:36:43.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:36:43.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:36:43.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:36:43.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:36:43.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:36:43.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:36:43.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:36:43.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:36:43.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:36:43.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:36:43.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:36:43.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:36:43.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:36:43.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:36:43.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:36:43.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:36:43.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:36:43.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:36:43.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:43.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:36:43.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:36:43.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:36:43.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:36:43.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:36:43.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:36:43.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:36:43.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:36:43.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:36:43.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:36:43.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:36:43.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:36:43.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:36:43.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:36:43.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:36:43.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:36:43.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:36:43.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:36:43.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:36:43.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:36:43.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:36:43.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:36:43.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:43.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:36:43.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:36:43.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:36:43.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:36:43.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:36:43.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:36:43.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:43.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:36:43.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176472 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:36:43.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:36:44.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:36:44.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:44.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:44.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:44.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:44.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:36:44.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:36:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:36:44.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:36:44.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:36:44.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:36:44.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:36:44.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:36:44.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:36:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:36:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:36:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:36:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:36:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:36:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176472/ceph-mon.a.asok 2024-04-25T07:36:44.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:36:44.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176472/ceph-mon.a.asok config get fsid 2024-04-25T07:36:44.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:36:44.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:36:44.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:36:44.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:36:44.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176472/ceph-mon.a.asok 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:36:44.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176472/ceph-mon.a.asok config get mon_host 2024-04-25T07:36:44.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2024-04-25T07:36:44.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:36:44.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:36:44.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:36:44.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:36:44.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:36:44.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:36:45.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:36:45.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:45.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:45.532 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:45.532 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:45.532 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:36:45.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:36:45.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:36:45.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:36:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2024-04-25T07:36:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:36:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:36:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:36:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:36:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:36:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:36:45.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:45.569 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:36:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:36:45.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:36:45.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=98cc7668-77f6-4d73-97e7-3f77c2a42683 2024-04-25T07:36:45.571 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 98cc7668-77f6-4d73-97e7-3f77c2a42683 2024-04-25T07:36:45.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 98cc7668-77f6-4d73-97e7-3f77c2a42683' 2024-04-25T07:36:45.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:36:45.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANCCpmAjsJIxAA8ByC5MA3uDEpakr4UmWYUQ== 2024-04-25T07:36:45.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANCCpmAjsJIxAA8ByC5MA3uDEpakr4UmWYUQ=="}' 2024-04-25T07:36:45.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 98cc7668-77f6-4d73-97e7-3f77c2a42683 -i td/osd-markdown/0/new.json 2024-04-25T07:36:45.921 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:36:45.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:36:45.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANCCpmAjsJIxAA8ByC5MA3uDEpakr4UmWYUQ== --osd-uuid 98cc7668-77f6-4d73-97e7-3f77c2a42683 2024-04-25T07:36:45.969 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:45.968+0000 7fecf97597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:45.998 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:45.996+0000 7fecf97597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:46.000 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:46.000+0000 7fecf97597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:46.000 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:46.000+0000 7fecf97597c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:36:46.000 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:46.000+0000 7fecf97597c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:36:46.000 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:46.000+0000 7fecf97597c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:36:46.038 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:46.036+0000 7fecf97597c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:36:52.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:36:52.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:36:52.278 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:36:52.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:36:52.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:36:52.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:36:52.841 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:36:52.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:52.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:36:52.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:36:52.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:36:52.865 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:52.864+0000 7f993b1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:52.867 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:52.868+0000 7f993b1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:52.869 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:52.868+0000 7f993b1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:53.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:36:53.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:53.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:36:53.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:53.283 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:36:53.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:53.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:53.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:53.440+0000 7f993b1cd7c0 -1 Falling back to public interface 2024-04-25T07:36:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:54.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:54.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:54.722 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:36:54.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:54.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:54.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:55.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:56.070 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:36:56.068+0000 7f993b1cd7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:36:56.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:56.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:56.148 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:36:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:36:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:56.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:57.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:57.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:57.796 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:36:57.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:36:57.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:57.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:58.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:59.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:59.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:59.256 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:36:59.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:36:59.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:59.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:59.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:00.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:00.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:00.665 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:37:00.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:37:00.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:00.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:01.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:02.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:02.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:02.256 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:37:02.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:37:02.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:02.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:02.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:03.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:03.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:03.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:37:03.701 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:37:03.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:03.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:04.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:05.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:05.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:05.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:37:05.239 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:37:05.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:05.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:05.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:06.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:06.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:06.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:37:06.704 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:37:06.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:06.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:07.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:08.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:08.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:08.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:37:08.201 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:37:08.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:08.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:08.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:09.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:09.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:09.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:37:09.631 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:37:09.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:09.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:10.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:11.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:11.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:11.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:37:11.966 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:37:11.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:11.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:12.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:13.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:13.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:13.500 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:37:13.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:37:13.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:13.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:14.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:15.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:15.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:15.338 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:37:15.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:37:15.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:15.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:16.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:17.117 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:37:17.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:37:17.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:17.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:17.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:18.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:18.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:18.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:37:18.995 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:37:18.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:18.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:19.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:20.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:20.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:20.950 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:37:20.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:37:20.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:20.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:21.442 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1517384320,v1:127.0.0.1:6803/1517384320] [v2:127.0.0.1:6804/1517384320,v1:127.0.0.1:6805/1517384320] exists,up 98cc7668-77f6-4d73-97e7-3f77c2a42683 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:21.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:37:21.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:21.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:21.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:37:21.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:37:21.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:37:21.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:37:21.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f260f637-e2cc-419f-b61e-560731589582 2024-04-25T07:37:21.449 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 f260f637-e2cc-419f-b61e-560731589582 2024-04-25T07:37:21.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f260f637-e2cc-419f-b61e-560731589582' 2024-04-25T07:37:21.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:37:21.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxCCpmNh7ZGxAAuIRm1feDGWmU1cmgQVXnbA== 2024-04-25T07:37:21.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxCCpmNh7ZGxAAuIRm1feDGWmU1cmgQVXnbA=="}' 2024-04-25T07:37:21.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f260f637-e2cc-419f-b61e-560731589582 -i td/osd-markdown/1/new.json 2024-04-25T07:37:22.001 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:37:22.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:37:22.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxCCpmNh7ZGxAAuIRm1feDGWmU1cmgQVXnbA== --osd-uuid f260f637-e2cc-419f-b61e-560731589582 2024-04-25T07:37:22.052 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:22.052+0000 7fbf9697f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:22.070 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:22.068+0000 7fbf9697f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:22.072+0000 7fbf9697f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:22.072+0000 7fbf9697f7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:37:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:22.072+0000 7fbf9697f7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:37:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:22.072+0000 7fbf9697f7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:37:22.110 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:22.108+0000 7fbf9697f7c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:37:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:37:28.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:37:28.110 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:37:28.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:37:28.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:37:28.671 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:37:28.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:37:28.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:37:28.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:37:28.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:37:28.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:37:28.702 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:28.699+0000 7f53dec947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:28.704 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:28.703+0000 7f53dec947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:28.705 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:28.703+0000 7f53dec947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:29.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:37:29.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:37:29.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:37:29.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:37:29.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:37:29.074 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:37:29.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:29.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:37:29.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:29.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:29.309 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:29.307+0000 7f53dec947c0 -1 Falling back to public interface 2024-04-25T07:37:29.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:30.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:30.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:30.535 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:37:30.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:37:30.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:30.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:30.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:31.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:31.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:31.940 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:37:31.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:37:31.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:31.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:31.960 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:31.959+0000 7f53dec947c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:37:32.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:33.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:33.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:33.347 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:37:33.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:33.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:33.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:33.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:34.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:34.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:34.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:37:34.860 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:37:34.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:34.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:35.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:36.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:36.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:36.310 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:37:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:37:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:36.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:36.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:37.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:37.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:37.884 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:37:37.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:37:37.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:37.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:38.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:39.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:39.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:39.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:37:39.376 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:37:39.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:39.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:39.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:40.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:40.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:40.865 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:37:40.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:37:40.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:40.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:41.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:42.317 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:37:42.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:37:42.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:42.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:42.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:43.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:43.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:43.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:37:43.865 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:37:43.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:43.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:44.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:45.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:45.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:45.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:37:45.651 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:37:45.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:45.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:46.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:47.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:47.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:47.441 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:37:47.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:37:47.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:47.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:48.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:49.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:49.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:49.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:37:49.100 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:37:49.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:49.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:50.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:51.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:51.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:51.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:37:51.051 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:37:51.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:51.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:51.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:52.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:52.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:52.736 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:37:52.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:37:52.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:52.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:54.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:54.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:54.676 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:37:54.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:37:54.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:54.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:55.111 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2884242288,v1:127.0.0.1:6811/2884242288] [v2:127.0.0.1:6812/2884242288,v1:127.0.0.1:6813/2884242288] exists,up f260f637-e2cc-419f-b61e-560731589582 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:37:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:55.115 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:37:55.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:37:55.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:37:55.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61f20fe7-0213-4390-bd1f-596e9174183d 2024-04-25T07:37:55.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 61f20fe7-0213-4390-bd1f-596e9174183d' 2024-04-25T07:37:55.118 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 61f20fe7-0213-4390-bd1f-596e9174183d 2024-04-25T07:37:55.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:37:55.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTCCpm+xo6CBAAaGYceEDMxt1Q72kMt64YfA== 2024-04-25T07:37:55.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTCCpm+xo6CBAAaGYceEDMxt1Q72kMt64YfA=="}' 2024-04-25T07:37:55.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61f20fe7-0213-4390-bd1f-596e9174183d -i td/osd-markdown/2/new.json 2024-04-25T07:37:55.739 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:37:55.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:37:55.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTCCpm+xo6CBAAaGYceEDMxt1Q72kMt64YfA== --osd-uuid 61f20fe7-0213-4390-bd1f-596e9174183d 2024-04-25T07:37:55.788 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:55.787+0000 7f13577b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:55.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:55.815+0000 7f13577b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:55.817 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:55.815+0000 7f13577b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:55.817 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:55.815+0000 7f13577b97c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:37:55.817 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:55.815+0000 7f13577b97c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:37:55.817 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:55.815+0000 7f13577b97c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:37:55.853 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:37:55.851+0000 7f13577b97c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:38:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:38:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:38:02.008 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:38:02.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:38:02.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:38:02.554 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:38:02.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:38:02.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:38:02.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:38:02.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:38:02.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:38:02.584 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:38:02.583+0000 7f33244f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:02.586 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:38:02.583+0000 7f33244f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:02.587 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:38:02.587+0000 7f33244f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:38:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:38:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:38:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:38:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:38:02.975 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:38:02.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:02.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:38:02.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:02.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:03.164 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:38:03.163+0000 7f33244f97c0 -1 Falling back to public interface 2024-04-25T07:38:03.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:04.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:04.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:04.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:38:04.375 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:38:04.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:04.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:04.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:05.675 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:38:05.671+0000 7f33244f97c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:38:05.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:05.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:05.842 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:38:05.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:38:05.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:05.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:06.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:07.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:07.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:07.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:38:07.485 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:38:07.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:07.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:07.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:08.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:08.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:08.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:38:08.952 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:38:08.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:08.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:09.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:10.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:10.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:10.417 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:38:10.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:38:10.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:10.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:10.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:11.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:11.929 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:38:11.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:11.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:38:11.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:11.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:13.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:13.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:13.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:38:13.413 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:38:13.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:13.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:13.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:14.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:14.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:14.907 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:38:14.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:38:14.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:14.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:15.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:16.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:16.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:16.351 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:38:16.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:38:16.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:16.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:16.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:17.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:17.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:17.934 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:38:17.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:38:17.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:17.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:18.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:19.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:19.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:19.422 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:38:19.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:38:19.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:19.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:20.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:21.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:21.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:21.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:38:21.211 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:38:21.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:21.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:21.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:22.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:22.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:22.933 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:38:22.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:38:22.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:22.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:23.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:24.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:24.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:24.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:38:24.666 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:38:24.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:24.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:25.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:26.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:26.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:26.558 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:38:26.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:38:26.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:26.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:27.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:28.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:28.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:28.042 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:38:28.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:38:28.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:28.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:28.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:29.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:29.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:29.593 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T07:38:29.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:38:29.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:29.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:30.065 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/398327419,v1:127.0.0.1:6819/398327419] [v2:127.0.0.1:6820/398327419,v1:127.0.0.1:6821/398327419] exists,up 61f20fe7-0213-4390-bd1f-596e9174183d 2024-04-25T07:38:30.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:38:30.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:38:30.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:38:30.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2024-04-25T07:38:30.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:38:30.516 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-04-25T07:38:30.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:38:30.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:38:31.040 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-04-25T07:38:31.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:38:32.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:38:33.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2024-04-25T07:38:33.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2024-04-25T07:38:33.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2024-04-25T07:38:33.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:38:34.119 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:38:34.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2024-04-25T07:38:34.290 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:38:34.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2024-04-25T07:38:34.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-04-25T07:38:34.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2024-04-25T07:38:34.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:38:34.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-04-25T07:38:34.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:38:34.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:38:34.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:38:34.841 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:38:34.841 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:38:34.841 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:38:34.841 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:38:34.841 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:38:34.842 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:38:34.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:38:34.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:38:34.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:38:35.273 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:38:35.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:38:35.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:38:35.695 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:38:35.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:38:45.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:38:45.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:38:45.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:38:46.277 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:38:46.278 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:38:46.278 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:38:46.278 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:38:46.278 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:38:46.278 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:38:46.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:38:46.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:38:46.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:38:46.761 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:38:46.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:38:46.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:38:47.208 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:38:47.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:38:57.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:38:57.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:38:57.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:38:57.768 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:38:57.768 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:38:57.768 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:38:57.768 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:38:57.768 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:38:57.768 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:38:57.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:38:57.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:38:57.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:38:58.239 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:38:58.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:38:58.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:38:58.726 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:38:58.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:39:08.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:39:08.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:39:08.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:39:09.288 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:39:09.288 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:39:09.289 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:39:09.289 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:39:09.289 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:39:09.289 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:39:09.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:39:09.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:39:09.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:39:09.721 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:39:09.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:39:09.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:39:10.153 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:39:10.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:39:20.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2024-04-25T07:39:35.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:39:35.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2024-04-25T07:39:35.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2024-04-25T07:39:35.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2024-04-25T07:39:35.778 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:39:35.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:39:35.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:39:35.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:39:35.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:39:35.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:39:35.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:39:35.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:39:35.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:39:35.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:39:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:39:36.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:39:36.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:39:36.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:39:36.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:39:36.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:39:36.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:39:36.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:39:36.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:39:36.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:39:36.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:39:36.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:39:36.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:39:36.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:39:36.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:39:36.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:36.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:39:36.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:39:36.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:39:36.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:39:36.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:39:36.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:39:36.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:39:36.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:39:36.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:39:36.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:39:36.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:39:36.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:39:36.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:39:36.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:39:36.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:39:36.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:39:36.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:39:36.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:39:36.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:36.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:39:36.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:39:36.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:39:36.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:39:36.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:39:36.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:39:36.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:36.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:39:36.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176472 2024-04-25T07:39:36.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:39:36.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:39:36.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:39:36.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:39:36.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2024-04-25T07:39:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2024-04-25T07:39:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2024-04-25T07:39:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:39:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:39:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:39:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:39:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:39:36.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:39:36.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:39:36.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:39:36.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:39:36.523 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:39:36.523 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:36.524 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:39:36.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:39:36.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:39:36.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:39:36.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:39:36.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:39:36.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:39:36.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:39:36.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:39:36.725 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:39:36.725 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:39:36.725 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:39:36.726 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:39:36.726 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:36.726 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:39:36.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176472/ceph-mon.a.asok 2024-04-25T07:39:36.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:39:36.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176472/ceph-mon.a.asok config get fsid 2024-04-25T07:39:36.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:39:36.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:39:36.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:39:36.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:39:36.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:39:36.844 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:39:36.844 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:39:36.844 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:39:36.845 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:39:36.845 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:36.845 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:39:36.845 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176472/ceph-mon.a.asok 2024-04-25T07:39:36.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:39:36.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176472/ceph-mon.a.asok config get mon_host 2024-04-25T07:39:36.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2024-04-25T07:39:36.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:39:36.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:39:36.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:39:36.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:39:36.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:39:36.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:39:37.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:39:37.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:39:37.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:39:37.860 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:39:37.860 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:37.860 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:39:37.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:39:37.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:39:37.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:39:37.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2024-04-25T07:39:37.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:39:37.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:39:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:39:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:39:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:39:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:39:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:39:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:39:37.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:39:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7340cb5c-a963-4b0f-b713-f581bab4e81a 2024-04-25T07:39:37.894 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 7340cb5c-a963-4b0f-b713-f581bab4e81a 2024-04-25T07:39:37.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7340cb5c-a963-4b0f-b713-f581bab4e81a' 2024-04-25T07:39:37.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:39:37.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5CCpmx2I5NhAA3yYSjDbecPQeU7tRdSiCdQ== 2024-04-25T07:39:37.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5CCpmx2I5NhAA3yYSjDbecPQeU7tRdSiCdQ=="}' 2024-04-25T07:39:37.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7340cb5c-a963-4b0f-b713-f581bab4e81a -i td/osd-markdown/0/new.json 2024-04-25T07:39:38.193 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:39:38.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:39:38.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5CCpmx2I5NhAA3yYSjDbecPQeU7tRdSiCdQ== --osd-uuid 7340cb5c-a963-4b0f-b713-f581bab4e81a 2024-04-25T07:39:38.241 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:38.241+0000 7f5fce0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:38.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:38.289+0000 7f5fce0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:38.289 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:38.289+0000 7f5fce0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:38.289 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:38.289+0000 7f5fce0b17c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:39:38.290 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:38.289+0000 7f5fce0b17c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:39:38.290 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:38.289+0000 7f5fce0b17c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:39:38.327 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:38.325+0000 7f5fce0b17c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:39:44.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:39:44.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:39:44.182 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:39:44.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:39:44.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:39:44.731 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:39:44.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:39:44.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:39:44.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:39:44.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:39:44.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:39:44.760 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:44.760+0000 7fcf8a6f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:44.776 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:44.776+0000 7fcf8a6f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:44.777 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:44.776+0000 7fcf8a6f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:45.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:39:45.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:39:45.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:39:45.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:39:45.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:39:45.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:45.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:39:45.112 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:39:45.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:45.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:45.317 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:45.316+0000 7fcf8a6f87c0 -1 Falling back to public interface 2024-04-25T07:39:45.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:46.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:46.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:46.574 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:39:46.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:39:46.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:46.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:46.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:47.831 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:39:47.832+0000 7fcf8a6f87c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:39:47.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:47.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:47.992 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:39:47.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:39:47.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:47.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:48.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:49.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:49.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:49.549 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:39:49.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:39:49.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:49.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:49.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:50.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:50.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:50.973 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:39:50.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:39:50.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:50.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:51.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:52.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:52.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:52.412 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:39:52.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:39:52.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:52.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:52.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:53.827 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:39:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:39:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:54.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:55.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:55.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:55.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:39:55.249 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:39:55.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:55.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:55.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:56.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:56.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:56.696 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:39:56.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:39:56.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:56.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:57.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:58.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:58.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:58.120 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:39:58.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:39:58.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:58.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:59.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:59.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:59.572 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:39:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:39:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:00.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:01.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:01.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:01.224 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:40:01.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:40:01.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:01.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:01.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:02.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:02.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:02.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:40:02.749 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:40:02.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:02.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:03.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:04.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:04.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:04.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:40:04.740 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:40:04.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:04.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:05.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:06.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:06.778 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:40:06.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:40:06.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:06.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:07.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:08.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:08.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:08.250 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:40:08.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:40:08.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:08.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:08.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:09.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:09.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:09.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:40:09.955 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:40:09.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:09.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:10.427 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1228009968,v1:127.0.0.1:6803/1228009968] [v2:127.0.0.1:6804/1228009968,v1:127.0.0.1:6805/1228009968] exists,up 7340cb5c-a963-4b0f-b713-f581bab4e81a 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:40:10.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:10.431 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:40:10.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:40:10.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:40:10.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4fa8ebb5-1b9b-4073-b195-10dfd41349db 2024-04-25T07:40:10.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4fa8ebb5-1b9b-4073-b195-10dfd41349db' 2024-04-25T07:40:10.433 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 4fa8ebb5-1b9b-4073-b195-10dfd41349db 2024-04-25T07:40:10.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:40:10.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaCCpmlBIKGxAAfLfrxcQvLjLtNxZwFZv8Zg== 2024-04-25T07:40:10.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaCCpmlBIKGxAAfLfrxcQvLjLtNxZwFZv8Zg=="}' 2024-04-25T07:40:10.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4fa8ebb5-1b9b-4073-b195-10dfd41349db -i td/osd-markdown/1/new.json 2024-04-25T07:40:10.946 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:40:10.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:40:10.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaCCpmlBIKGxAAfLfrxcQvLjLtNxZwFZv8Zg== --osd-uuid 4fa8ebb5-1b9b-4073-b195-10dfd41349db 2024-04-25T07:40:10.997 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:10.996+0000 7f889b28e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:11.001 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:11.000+0000 7f889b28e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:11.002 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:11.000+0000 7f889b28e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:11.003 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:11.004+0000 7f889b28e7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:40:11.003 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:11.004+0000 7f889b28e7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:40:11.003 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:11.004+0000 7f889b28e7c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:40:11.047 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:11.048+0000 7f889b28e7c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:40:16.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:40:16.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:40:16.850 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:40:16.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:40:16.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:40:17.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:40:17.397 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:40:17.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:40:17.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:40:17.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:17.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:40:17.427 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:17.424+0000 7f6c895fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:17.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:17.452+0000 7f6c895fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:17.453 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:17.452+0000 7f6c895fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:17.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:40:17.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:17.826 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:40:17.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:40:17.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:17.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:17.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:17.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:17.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:17.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:18.036 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:18.036+0000 7f6c895fb7c0 -1 Falling back to public interface 2024-04-25T07:40:18.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:19.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:19.222 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:40:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:19.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:19.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:19.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:20.564 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:20.564+0000 7f6c895fb7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:40:20.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:20.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:20.625 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:40:20.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:20.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:20.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:21.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:22.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:22.262 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:40:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:22.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:22.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:23.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:23.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:23.706 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:40:23.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:40:23.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:23.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:24.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:25.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:25.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:25.147 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:40:25.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:40:25.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:25.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:25.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:26.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:26.613 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:40:26.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:40:26.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:26.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:27.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:28.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:28.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:28.027 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:40:28.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:40:28.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:28.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:28.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:29.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:29.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:29.535 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:40:29.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:40:29.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:29.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:29.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:30.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:30.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:30.953 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:40:30.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:40:30.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:30.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:31.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:32.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:32.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:32.632 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:40:32.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:40:32.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:32.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:33.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:34.538 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:40:34.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:40:34.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:34.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:34.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:35.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:35.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:35.980 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:40:35.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:40:35.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:35.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:36.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:37.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:37.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:37.729 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:40:37.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:40:37.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:37.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:38.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:39.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:39.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:39.479 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:40:39.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:40:39.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:39.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:40.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:41.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:41.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:41.320 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:40:41.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:40:41.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:41.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:41.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:42.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:42.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:42.983 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:40:42.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:40:42.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:42.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:43.536 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4271417476,v1:127.0.0.1:6811/4271417476] [v2:127.0.0.1:6812/4271417476,v1:127.0.0.1:6813/4271417476] exists,up 4fa8ebb5-1b9b-4073-b195-10dfd41349db 2024-04-25T07:40:43.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:43.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:43.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:43.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2024-04-25T07:40:43.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:40:43.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:40:43.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:40:43.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:43.540 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:40:43.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:40:43.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:40:43.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:40:43.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9553b3bd-83c2-4413-b3ad-a156a90e7a85 2024-04-25T07:40:43.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9553b3bd-83c2-4413-b3ad-a156a90e7a85' 2024-04-25T07:40:43.543 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 9553b3bd-83c2-4413-b3ad-a156a90e7a85 2024-04-25T07:40:43.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:40:43.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7CCpmGmFmIRAAYZak95Jl/g+kaHi/uO14+w== 2024-04-25T07:40:43.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7CCpmGmFmIRAAYZak95Jl/g+kaHi/uO14+w=="}' 2024-04-25T07:40:43.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9553b3bd-83c2-4413-b3ad-a156a90e7a85 -i td/osd-markdown/2/new.json 2024-04-25T07:40:44.166 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:40:44.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:40:44.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7CCpmGmFmIRAAYZak95Jl/g+kaHi/uO14+w== --osd-uuid 9553b3bd-83c2-4413-b3ad-a156a90e7a85 2024-04-25T07:40:44.215 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:44.215+0000 7f7bd6b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:44.217 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:44.215+0000 7f7bd6b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:44.218 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:44.215+0000 7f7bd6b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:44.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:44.219+0000 7f7bd6b4d7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:40:44.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:44.219+0000 7f7bd6b4d7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:40:44.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:44.219+0000 7f7bd6b4d7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:40:44.298 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:44.295+0000 7f7bd6b4d7c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:40:50.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:40:50.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:40:50.388 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:40:50.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:40:50.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:40:50.955 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:40:50.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:40:50.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:50.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:40:50.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:40:50.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:40:50.978 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:50.975+0000 7fee545757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:50.984 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:50.983+0000 7fee545757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:50.986 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:50.983+0000 7fee545757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:40:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:40:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:51.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:51.348 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:40:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:51.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:51.533 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:51.531+0000 7fee545757c0 -1 Falling back to public interface 2024-04-25T07:40:51.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:52.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:52.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:52.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:52.765 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:40:52.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:52.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:53.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:54.018 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:40:54.015+0000 7fee545757c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:40:54.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:54.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:54.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:54.173 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:40:54.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:54.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:54.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:55.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:55.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:55.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:55.833 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:40:55.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:55.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:56.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:57.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:57.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:57.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:40:57.287 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:40:57.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:57.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:57.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:58.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:58.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:58.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:40:58.752 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:40:58.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:58.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:59.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:00.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:00.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:00.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:41:00.245 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:41:00.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:00.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:01.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:01.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:01.653 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:41:01.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:41:01.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:01.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:02.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:03.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:03.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:03.075 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:41:03.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:41:03.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:03.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:03.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:04.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:04.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:04.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:41:04.573 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:41:04.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:04.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:05.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:06.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:06.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:06.234 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:41:06.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:41:06.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:06.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:06.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:07.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:07.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:07.651 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:41:07.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:41:07.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:07.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:08.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:09.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:09.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:09.446 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:41:09.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:41:09.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:09.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:10.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:11.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:11.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:11.079 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:41:11.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:41:11.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:11.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:11.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:12.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:12.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:12.865 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:41:12.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:41:12.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:12.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:13.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:14.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:14.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:14.629 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:41:14.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:41:14.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:14.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:15.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:16.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:16.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:16.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:41:16.346 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:41:16.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:16.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:41:16.907 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/605633564,v1:127.0.0.1:6819/605633564] [v2:127.0.0.1:6820/605633564,v1:127.0.0.1:6821/605633564] exists,up 9553b3bd-83c2-4413-b3ad-a156a90e7a85 2024-04-25T07:41:16.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:41:16.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:41:16.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:41:16.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2024-04-25T07:41:16.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:41:17.345 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-04-25T07:41:17.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:41:17.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:41:17.869 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-04-25T07:41:17.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:41:18.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:41:20.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2024-04-25T07:41:20.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2024-04-25T07:41:20.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2024-04-25T07:41:20.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:41:20.396 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:41:20.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2024-04-25T07:41:20.577 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T07:41:20.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2024-04-25T07:41:20.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-04-25T07:41:20.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2024-04-25T07:41:20.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:41:20.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-04-25T07:41:20.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:41:20.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:41:20.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:41:21.159 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:41:21.159 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:41:21.159 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:41:21.159 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:41:21.159 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:41:21.159 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:41:21.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:41:21.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:41:21.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:41:21.558 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:41:21.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:41:21.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:41:21.979 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:41:22.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:41:32.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:41:32.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:41:32.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:41:32.561 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:41:32.561 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:41:32.562 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:41:32.562 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:41:32.562 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:41:32.562 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:41:32.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:41:32.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:41:32.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:41:32.990 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:41:32.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:41:32.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:41:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:41:33.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:41:43.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:41:43.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:41:43.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:41:43.961 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:41:43.962 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:41:43.962 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:41:43.962 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:41:43.962 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:41:43.962 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:41:43.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:41:43.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:41:43.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:41:44.397 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:41:44.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:41:44.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:41:44.811 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:41:44.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:41:54.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:41:54.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:41:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:41:55.413 INFO:tasks.workunit.client.0.smithi088.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:41:55.413 INFO:tasks.workunit.client.0.smithi088.stdout:-1 0.29306 root default 2024-04-25T07:41:55.413 INFO:tasks.workunit.client.0.smithi088.stdout:-3 0.29306 host smithi088 2024-04-25T07:41:55.413 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:41:55.413 INFO:tasks.workunit.client.0.smithi088.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:41:55.413 INFO:tasks.workunit.client.0.smithi088.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:41:55.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:41:55.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:41:55.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:41:55.852 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:41:55.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:41:55.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:41:56.258 INFO:tasks.workunit.client.0.smithi088.stderr:marked down osd.0. 2024-04-25T07:41:56.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:42:06.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2024-04-25T07:42:06.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2024-04-25T07:42:06.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2024-04-25T07:42:06.688 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-04-25T07:42:06.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:42:06.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:42:06.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:42:06.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:42:06.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:06.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:06.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:06.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:06.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:06.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:06.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:06.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:06.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:06.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:06.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:06.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:06.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:06.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:06.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:06.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:06.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:06.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:42:06.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:42:06.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:06.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:06.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:06.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:42:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:42:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:42:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:06.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:06.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:06.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:06.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:06.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:06.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:06.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:06.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:06.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:06.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:06.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:42:06.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:42:06.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:06.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:06.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:06.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:42:06.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:06.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:06.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:42:06.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:42:06.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:06.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:06.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.176472 2024-04-25T07:42:06.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2024-04-25T07:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2024-04-25T07:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2024-04-25T07:42:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:42:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:42:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:42:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:42:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:42:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:42:07.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:42:07.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:07.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:07.291 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:07.291 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:07.291 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:07.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:42:07.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:42:07.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:42:07.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:42:07.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:42:07.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:42:07.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:42:07.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:42:07.500 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:42:07.500 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:42:07.501 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:42:07.501 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:42:07.501 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:07.501 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:07.501 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176472/ceph-mon.a.asok 2024-04-25T07:42:07.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:42:07.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176472/ceph-mon.a.asok config get fsid 2024-04-25T07:42:07.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:42:07.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:42:07.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:42:07.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:42:07.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:42:07.618 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:42:07.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:42:07.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:42:07.619 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:42:07.619 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:07.619 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:07.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.176472/ceph-mon.a.asok 2024-04-25T07:42:07.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:42:07.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.176472/ceph-mon.a.asok config get mon_host 2024-04-25T07:42:07.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2024-04-25T07:42:07.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:42:07.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:42:07.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:42:07.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:42:07.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:42:07.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:42:08.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:42:08.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:08.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:08.744 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:08.744 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:08.744 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:08.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:42:08.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:42:08.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:42:08.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2024-04-25T07:42:08.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:42:08.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:42:08.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:42:08.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:42:08.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:42:08.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:42:08.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:42:08.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:42:08.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:42:08.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:42:08.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:42:08.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=09524a2c-830f-4fc3-a650-dc18ab2749e0 2024-04-25T07:42:08.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 09524a2c-830f-4fc3-a650-dc18ab2749e0' 2024-04-25T07:42:08.812 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 09524a2c-830f-4fc3-a650-dc18ab2749e0 2024-04-25T07:42:08.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:42:08.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQCSpmRwJkMRAAJTcw6JFe0nE0PN5NNx8T5g== 2024-04-25T07:42:08.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQCSpmRwJkMRAAJTcw6JFe0nE0PN5NNx8T5g=="}' 2024-04-25T07:42:08.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 09524a2c-830f-4fc3-a650-dc18ab2749e0 -i td/osd-markdown/0/new.json 2024-04-25T07:42:09.103 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:42:09.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:42:09.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQCSpmRwJkMRAAJTcw6JFe0nE0PN5NNx8T5g== --osd-uuid 09524a2c-830f-4fc3-a650-dc18ab2749e0 2024-04-25T07:42:09.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:09.150+0000 7f179cc867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:09.155 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:09.158+0000 7f179cc867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:09.158+0000 7f179cc867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:09.158+0000 7f179cc867c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:42:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:09.158+0000 7f179cc867c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:42:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:09.158+0000 7f179cc867c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:42:09.195 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:09.198+0000 7f179cc867c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:42:15.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:42:15.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:42:15.123 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:42:15.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:42:15.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:42:15.702 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:42:15.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:42:15.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:42:15.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:42:15.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:42:15.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:42:15.732 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:15.733+0000 7ff20ae097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:15.733 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:15.733+0000 7ff20ae097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:15.735 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:15.733+0000 7ff20ae097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:42:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:42:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:42:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:42:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:42:16.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:16.144 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:42:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:42:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:16.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:16.328 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:16.329+0000 7ff20ae097c0 -1 Falling back to public interface 2024-04-25T07:42:16.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:17.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:17.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:17.568 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:42:17.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:42:17.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:17.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:17.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:18.958 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:18.957+0000 7ff20ae097c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:42:18.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:18.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:18.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:42:18.974 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:42:18.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:18.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:19.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:20.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:20.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:20.604 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:42:20.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:42:20.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:20.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:21.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:22.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:22.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:22.032 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:42:22.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:42:22.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:22.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:22.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:23.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:23.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:23.472 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:42:23.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:42:23.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:23.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:24.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:25.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:25.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:25.029 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:42:25.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:42:25.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:25.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:25.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:26.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:26.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:26.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:42:26.436 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:42:26.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:26.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:27.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:28.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:28.016 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:42:28.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:28.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:42:28.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:28.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:28.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:29.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:42:29.456 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:42:29.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:29.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:29.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:30.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:30.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:30.962 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:42:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:42:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:31.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:32.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:32.488 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:42:32.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:32.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:42:32.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:32.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:33.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:34.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:34.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:34.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:42:34.413 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:42:34.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:34.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:35.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:36.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:36.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:36.223 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:42:36.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:42:36.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:36.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:37.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:38.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:38.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:38.194 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:42:38.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:42:38.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:38.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:39.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:40.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:40.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:40.180 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:42:40.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:42:40.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:40.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:40.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:41.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:41.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:41.721 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:42:41.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:42:41.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:41.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:42.129 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3482652395,v1:127.0.0.1:6803/3482652395] [v2:127.0.0.1:6804/3482652395,v1:127.0.0.1:6805/3482652395] exists,up 09524a2c-830f-4fc3-a650-dc18ab2749e0 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:42:42.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:42:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:42:42.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:42:42.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da8c57b2-be6b-4b7b-9d1f-a2ffd715be9e 2024-04-25T07:42:42.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 da8c57b2-be6b-4b7b-9d1f-a2ffd715be9e' 2024-04-25T07:42:42.135 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 da8c57b2-be6b-4b7b-9d1f-a2ffd715be9e 2024-04-25T07:42:42.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:42:42.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByCSpmqP9cCRAAtq45LtsYwaBWDL1ndtavbg== 2024-04-25T07:42:42.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByCSpmqP9cCRAAtq45LtsYwaBWDL1ndtavbg=="}' 2024-04-25T07:42:42.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da8c57b2-be6b-4b7b-9d1f-a2ffd715be9e -i td/osd-markdown/1/new.json 2024-04-25T07:42:42.604 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:42:42.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:42:42.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByCSpmqP9cCRAAtq45LtsYwaBWDL1ndtavbg== --osd-uuid da8c57b2-be6b-4b7b-9d1f-a2ffd715be9e 2024-04-25T07:42:42.647 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:42.649+0000 7f605a6d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:42.665 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:42.665+0000 7f605a6d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:42.666 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:42.665+0000 7f605a6d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:42.667 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:42.669+0000 7f605a6d57c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:42:42.667 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:42.669+0000 7f605a6d57c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:42:42.667 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:42.669+0000 7f605a6d57c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:42:42.705 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:42.705+0000 7f605a6d57c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:42:48.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:42:48.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:42:48.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:42:48.552 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:42:48.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:42:49.094 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:42:49.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:42:49.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:42:49.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:42:49.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:42:49.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:42:49.124 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:49.125+0000 7ff6cdd107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:49.152 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:49.153+0000 7ff6cdd107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:49.153 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:49.153+0000 7ff6cdd107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:49.709 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:49.709+0000 7ff6cdd107c0 -1 Falling back to public interface 2024-04-25T07:42:49.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:42:49.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:42:49.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:42:49.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:42:49.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:42:49.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:49.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:42:49.809 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:42:49.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:49.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:42:50.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:51.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:51.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:51.231 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:42:51.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:42:51.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:51.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:42:51.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:52.381 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:42:52.381+0000 7ff6cdd107c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:42:52.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:52.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:52.663 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:42:52.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:42:52.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:52.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:42:53.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:54.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:54.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:54.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:42:54.164 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:42:54.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:54.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:42:54.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:55.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:55.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:55.587 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:42:55.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:42:55.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:55.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:42:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:57.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:57.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:57.151 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:42:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:42:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:42:57.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:58.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:58.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:58.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:42:58.550 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:42:58.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:58.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:42:59.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:00.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:00.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:00.003 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:43:00.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:43:00.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:00.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:00.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:01.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:01.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:01.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:43:01.486 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:43:01.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:01.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:01.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:02.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:02.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:02.953 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:43:02.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:43:02.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:02.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:03.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:04.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:04.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:04.531 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:43:04.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:43:04.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:04.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:05.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:06.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:06.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:06.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:43:06.005 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:43:06.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:06.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:07.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:08.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:08.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:08.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:43:08.029 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:43:08.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:08.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:08.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:09.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:09.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:09.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:43:09.983 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:43:09.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:09.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:12.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:12.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:12.103 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:43:12.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:43:12.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:12.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:13.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:14.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:14.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:14.078 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:43:14.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:43:14.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:14.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:14.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:15.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:15.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:15.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:43:15.573 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:43:15.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:15.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:43:15.978 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/92628084,v1:127.0.0.1:6811/92628084] [v2:127.0.0.1:6812/92628084,v1:127.0.0.1:6813/92628084] exists,up da8c57b2-be6b-4b7b-9d1f-a2ffd715be9e 2024-04-25T07:43:15.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:43:15.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:43:15.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:43:15.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2024-04-25T07:43:15.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:43:15.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:43:15.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:43:15.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:43:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:43:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:43:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:43:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:43:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:43:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:43:15.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:43:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f83551fe-4d22-4b4c-9f6d-bac5977954bc 2024-04-25T07:43:15.985 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 f83551fe-4d22-4b4c-9f6d-bac5977954bc 2024-04-25T07:43:15.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f83551fe-4d22-4b4c-9f6d-bac5977954bc' 2024-04-25T07:43:15.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:43:15.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTCSpmwKiEOxAA09hx67K/9J1At3dWdjt2Aw== 2024-04-25T07:43:15.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTCSpmwKiEOxAA09hx67K/9J1At3dWdjt2Aw=="}' 2024-04-25T07:43:15.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f83551fe-4d22-4b4c-9f6d-bac5977954bc -i td/osd-markdown/2/new.json 2024-04-25T07:43:16.523 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:43:16.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:43:16.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTCSpmwKiEOxAA09hx67K/9J1At3dWdjt2Aw== --osd-uuid f83551fe-4d22-4b4c-9f6d-bac5977954bc 2024-04-25T07:43:16.572 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:16.572+0000 7f58541d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:16.574 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:16.572+0000 7f58541d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:16.575 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:16.576+0000 7f58541d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:16.576 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:16.576+0000 7f58541d67c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:43:16.576 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:16.576+0000 7f58541d67c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:43:16.576 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:16.576+0000 7f58541d67c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:43:16.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:16.636+0000 7f58541d67c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:43:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:43:22.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:43:22.532 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:43:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:43:22.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:43:23.083 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:43:23.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:43:23.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:43:23.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:43:23.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:43:23.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6952c312-6baa-4992-a47c-47df5f6a0ab4 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.176472/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:43:23.113 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:23.112+0000 7f270b3687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:23.115 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:23.116+0000 7f270b3687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:23.116 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:23.116+0000 7f270b3687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:23.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:43:23.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:43:23.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:43:23.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:43:23.528 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:43:23.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:43:23.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:23.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:43:23.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:23.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:23.668 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:23.668+0000 7f270b3687c0 -1 Falling back to public interface 2024-04-25T07:43:23.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:24.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:24.974 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:43:24.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:24.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:43:24.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:24.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:25.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:26.328 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:43:26.328+0000 7f270b3687c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:43:26.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:26.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:26.457 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:43:26.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:43:26.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:26.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:26.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:27.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:27.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:27.977 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:43:27.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:43:27.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:27.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:28.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:29.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:29.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:29.419 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:43:29.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:43:29.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:29.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:29.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:30.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:30.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:30.876 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:43:30.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:43:30.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:30.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:31.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:32.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:32.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:32.380 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:43:32.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:43:32.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:32.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:32.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:33.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:33.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:33.848 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:43:33.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:43:33.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:33.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:34.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:35.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:35.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:35.463 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:43:35.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:43:35.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:35.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:35.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:36.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:36.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:36.893 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:43:36.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:43:36.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:36.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:37.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:38.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:38.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:38.390 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:43:38.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:43:38.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:38.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:38.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:39.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:39.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:39.951 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:43:39.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:43:39.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:39.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:40.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:41.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:41.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:41.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:43:41.786 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:43:41.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:41.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:42.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:43.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:43.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:43.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:43:43.598 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:43:43.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:43.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:44.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:45.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:45.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:45.314 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:43:45.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:43:45.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:45.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:46.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:47.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:47.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:47.241 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:43:47.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:43:47.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:47.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:48.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:49.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:49.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:49.192 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:43:49.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:43:49.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:49.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:43:49.609 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1394225653,v1:127.0.0.1:6819/1394225653] [v2:127.0.0.1:6820/1394225653,v1:127.0.0.1:6821/1394225653] exists,up f83551fe-4d22-4b4c-9f6d-bac5977954bc 2024-04-25T07:43:49.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:43:49.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:43:49.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:43:49.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2024-04-25T07:43:49.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=195456 2024-04-25T07:43:49.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 195456 2024-04-25T07:43:49.617 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2024-04-25T07:43:49.617 INFO:tasks.workunit.client.0.smithi088.stdout: 195456 ? 00:00:00 ceph-osd 2024-04-25T07:43:49.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2024-04-25T07:43:49.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2024-04-25T07:43:49.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2024-04-25T07:43:50.020 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:43:50.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2024-04-25T07:43:50.530 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already stopped. 2024-04-25T07:43:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2024-04-25T07:44:05.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2024-04-25T07:44:05.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2024-04-25T07:44:05.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2024-04-25T07:44:05.988 INFO:tasks.workunit.client.0.smithi088.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-04-25T07:44:05.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 195456 2024-04-25T07:44:05.997 INFO:tasks.workunit.client.0.smithi088.stdout: PID TTY TIME CMD 2024-04-25T07:44:05.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:44:05.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:44:05.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:05.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:44:05.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:05.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:05.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:05.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:05.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:06.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:06.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:06.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:06.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:06.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:06.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:06.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:06.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:06.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:06.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:06.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:06.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:06.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:06.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:44:06.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:06.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:06.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:44:06.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-markdown 0 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:06.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:06.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:06.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:06.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:06.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:06.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:06.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:06.132 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:06.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:06.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:06.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:06.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:06.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:06.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:44:06.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:44:06.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:06.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:06.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.176472 2024-04-25T07:44:06.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.176472 2024-04-25T07:44:06.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:06.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:06.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:44:06.138 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:44:06.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:44:06.188 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2024-04-25T07:44:06.188 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2024-04-25T07:44:06.242 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:44:06.247 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:44:06.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:44:06.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:44:06.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:44:06.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:44:06.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:44:06.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:44:06.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-reactivate 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=9da9f10e-9239-43aa-b12c-50a4ed9c7524 --auth-supported=none ' 2024-04-25T07:44:06.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2024-04-25T07:44:06.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2024-04-25T07:44:06.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:44:06.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:06.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:06.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:06.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:06.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:06.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:06.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:06.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:06.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:06.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:06.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:06.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:06.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:06.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:06.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:06.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T07:44:06.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:06.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:06.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:06.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.199893 2024-04-25T07:44:06.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:06.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:06.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2024-04-25T07:44:06.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:44:06.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:06.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:06.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.199893 2024-04-25T07:44:06.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:44:06.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:44:06.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:44:06.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:44:06.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2024-04-25T07:44:06.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2024-04-25T07:44:06.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2024-04-25T07:44:06.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2024-04-25T07:44:06.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2024-04-25T07:44:06.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:44:06.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:44:06.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:44:06.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2024-04-25T07:44:06.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2024-04-25T07:44:06.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:44:06.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:06.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:06.723 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:06.723 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:06.723 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:06.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.199893/$cluster-$name.asok' 2024-04-25T07:44:06.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.199893/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:44:06.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:44:06.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:44:06.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:06.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:06.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:44:06.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:44:06.928 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:06.928 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:06.928 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:06.929 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:06.929 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:06.929 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:06.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.199893/ceph-mon.a.asok 2024-04-25T07:44:06.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:06.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.199893/ceph-mon.a.asok config get fsid 2024-04-25T07:44:07.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:44:07.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:07.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:07.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:44:07.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:44:07.045 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:07.045 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:07.045 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:07.046 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:07.046 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:07.046 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:07.047 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.199893/ceph-mon.a.asok 2024-04-25T07:44:07.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:07.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.199893/ceph-mon.a.asok config get mon_host 2024-04-25T07:44:07.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2024-04-25T07:44:07.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2024-04-25T07:44:07.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:44:07.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:44:07.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:44:07.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2024-04-25T07:44:07.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:44:08.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:44:08.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:08.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:08.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:08.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:08.039 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:08.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.199893/$cluster-$name.asok' 2024-04-25T07:44:08.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:08.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.199893/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:08.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2024-04-25T07:44:08.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9da9f10e-9239-43aa-b12c-50a4ed9c7524 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-04-25T07:44:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.199893/$cluster-$name.asok' 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.199893/$cluster-$name.asok' 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2024-04-25T07:44:08.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:44:08.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fae91b26-19ef-498c-b924-e6e160506af2 2024-04-25T07:44:08.081 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 fae91b26-19ef-498c-b924-e6e160506af2 2024-04-25T07:44:08.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fae91b26-19ef-498c-b924-e6e160506af2' 2024-04-25T07:44:08.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:44:08.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDICSpmXAPLBRAASbhHgVnf4Jqp9b7GTqFNsg== 2024-04-25T07:44:08.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDICSpmXAPLBRAASbhHgVnf4Jqp9b7GTqFNsg=="}' 2024-04-25T07:44:08.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fae91b26-19ef-498c-b924-e6e160506af2 -i td/osd-reactivate/0/new.json 2024-04-25T07:44:08.379 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:44:08.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2024-04-25T07:44:08.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9da9f10e-9239-43aa-b12c-50a4ed9c7524 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.199893/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDICSpmXAPLBRAASbhHgVnf4Jqp9b7GTqFNsg== --osd-uuid fae91b26-19ef-498c-b924-e6e160506af2 2024-04-25T07:44:08.425 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:08.423+0000 7fc1cfefe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:08.439 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:08.439+0000 7fc1cfefe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:08.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:08.439+0000 7fc1cfefe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:08.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:08.439+0000 7fc1cfefe7c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T07:44:08.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:08.439+0000 7fc1cfefe7c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T07:44:08.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:08.439+0000 7fc1cfefe7c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T07:44:08.479 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:08.479+0000 7fc1cfefe7c0 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2024-04-25T07:44:14.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2024-04-25T07:44:14.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:44:14.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:44:14.333 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:44:14.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:44:14.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:44:14.910 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:44:14.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9da9f10e-9239-43aa-b12c-50a4ed9c7524 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.199893/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:44:14.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:44:14.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:44:14.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:44:14.933 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:14.931+0000 7f4c8a2d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:14.970 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:14.967+0000 7f4c8a2d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:14.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:14.971+0000 7f4c8a2d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:15.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:44:15.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:44:15.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:44:15.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:44:15.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:44:15.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:15.308 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:44:15.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:44:15.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:15.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:15.529 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:15.527+0000 7f4c8a2d47c0 -1 Falling back to public interface 2024-04-25T07:44:15.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:16.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:16.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:16.769 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:44:16.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:44:16.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:16.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:17.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:17.802 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:17.799+0000 7f4c8a2d47c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:44:18.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:18.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:18.217 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:44:18.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:44:18.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:18.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:18.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:19.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:19.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:19.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:44:19.907 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:44:19.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:19.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:20.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:21.325 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:44:21.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:44:21.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:21.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:21.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:22.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:22.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:22.838 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T07:44:22.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:44:22.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:22.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:23.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:24.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:24.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:24.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:44:24.274 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T07:44:24.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:24.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:24.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:25.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:25.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:25.698 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T07:44:25.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:44:25.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:25.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:26.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:27.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:27.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:27.126 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T07:44:27.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:44:27.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:27.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:27.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:28.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:28.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:28.691 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T07:44:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:44:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:29.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:30.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:30.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:30.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:44:30.096 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T07:44:30.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:30.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:30.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:31.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:31.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:31.572 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T07:44:31.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:44:31.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:31.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:31.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:32.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:32.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:32.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:44:32.990 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T07:44:32.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:32.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:33.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:34.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:34.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:34.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:44:34.910 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T07:44:34.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:34.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:35.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:36.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:36.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:36.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:44:36.702 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T07:44:36.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:36.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:37.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:38.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:38.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:38.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:44:38.527 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T07:44:38.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:38.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:39.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:40.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:40.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:40.456 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T07:44:40.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:44:40.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:40.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:40.930 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3633178852,v1:127.0.0.1:6803/3633178852] [v2:127.0.0.1:6804/3633178852,v1:127.0.0.1:6805/3633178852] exists,up fae91b26-19ef-498c-b924-e6e160506af2 2024-04-25T07:44:40.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:44:40.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:44:40.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:44:40.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2024-04-25T07:44:40.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:40.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:40.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:40.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:40.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2024-04-25T07:44:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2024-04-25T07:44:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2024-04-25T07:44:41.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-reactivate 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-reactivate/0 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9da9f10e-9239-43aa-b12c-50a4ed9c7524 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:44:41.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-04-25T07:44:41.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:44:41.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:41.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:41.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:41.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:41.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:41.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.199893/$cluster-$name.asok' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.199893/$cluster-$name.asok' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:44:41.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-reactivate/0 2024-04-25T07:44:41.247 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:44:41.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:44:41.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-reactivate/0/whoami 2024-04-25T07:44:41.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=9da9f10e-9239-43aa-b12c-50a4ed9c7524 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.199893/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:44:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:44:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:44:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:44:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:44:41.278 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:41.275+0000 7f393af747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:41.289 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:41.287+0000 7f393af747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:41.290 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:41.287+0000 7f393af747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:41.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:44:41.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:44:41.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:44:41.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:44:41.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:44:41.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:41.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:44:41.668 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:44:41.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:41.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:41.850 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:41.847+0000 7f393af747c0 -1 Falling back to public interface 2024-04-25T07:44:42.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:43.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:43.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:43.136 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:44:43.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:44:43.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:43.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:43.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:44.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:44.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:44.557 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:44:44.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:44:44.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:44.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:44.810 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:44.807+0000 7f393af747c0 -1 osd.0 5 log_to_monitors true 2024-04-25T07:44:45.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:46.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:46.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:46.008 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:44:46.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:44:46.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:46.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:46.418 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/4201390710,v1:127.0.0.1:6803/4201390710] [v2:127.0.0.1:6804/4201390710,v1:127.0.0.1:6805/4201390710] exists,up fae91b26-19ef-498c-b924-e6e160506af2 2024-04-25T07:44:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:44:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:44:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:44:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2024-04-25T07:44:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T07:44:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T07:44:46.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:46.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:46.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:46.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:46.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:46.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:46.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:46.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:46.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:46.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:46.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:46.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:46.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:46.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:46.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T07:44:46.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:46.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:46.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:46.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.199893 2024-04-25T07:44:46.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-reactivate 0 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:46.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:46.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:46.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:46.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:46.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:46.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:46.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:46.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:46.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:46.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:46.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:44:46.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T07:44:46.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:46.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:46.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.199893 2024-04-25T07:44:46.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.199893 2024-04-25T07:44:46.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:46.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:46.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:44:46.757 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:44:46.757 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:44:46.809 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2024-04-25T07:44:46.810 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2024-04-25T07:44:46.862 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-prio 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2024-04-25T07:44:46.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2024-04-25T07:44:46.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2024-04-25T07:44:46.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2024-04-25T07:44:46.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2024-04-25T07:44:46.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2024-04-25T07:44:46.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none ' 2024-04-25T07:44:46.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2024-04-25T07:44:46.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-04-25T07:44:46.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export objects=200 2024-04-25T07:44:46.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: objects=200 2024-04-25T07:44:46.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: export poolprefix=test 2024-04-25T07:44:46.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: poolprefix=test 2024-04-25T07:44:46.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: export FORCE_PRIO=255 2024-04-25T07:44:46.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: FORCE_PRIO=255 2024-04-25T07:44:46.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: export NORMAL_PRIO=190 2024-04-25T07:44:46.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: NORMAL_PRIO=190 2024-04-25T07:44:46.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: set 2024-04-25T07:44:46.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:44:46.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: local 'funcs=TEST_recovery_pool_priority 2024-04-25T07:44:46.872 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_recovery_priority' 2024-04-25T07:44:46.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:46.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:46.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:46.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:46.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:46.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:46.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:46.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:46.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:46.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:46.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:46.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:46.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:46.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T07:44:46.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:46.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:46.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:44:46.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202479 2024-04-25T07:44:46.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:46.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:46.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-04-25T07:44:46.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:44:46.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:46.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:44:46.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.202479 2024-04-25T07:44:46.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:44:46.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_pool_priority td/osd-recovery-prio 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:326: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: local pools=3 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: local OSDS=2 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:330: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:44:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:44:46.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-04-25T07:44:46.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-04-25T07:44:47.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:44:47.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:47.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:47.399 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:47.400 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:47.400 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:44:47.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:44:47.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:44:47.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:44:47.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:44:47.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:47.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:47.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:44:47.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:44:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:44:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-mon.a.asok 2024-04-25T07:44:47.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:47.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202479/ceph-mon.a.asok config get fsid 2024-04-25T07:44:47.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:44:47.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:47.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:47.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:44:47.728 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:47.728 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:47.728 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:47.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:44:47.729 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:47.729 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:47.729 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:44:47.729 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-mon.a.asok 2024-04-25T07:44:47.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:47.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202479/ceph-mon.a.asok config get mon_host 2024-04-25T07:44:47.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2024-04-25T07:44:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-04-25T07:44:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:44:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:44:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:44:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-04-25T07:44:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:44:48.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:44:48.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:48.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:48.760 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:48.760 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:48.760 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:44:48.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:44:48.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:48.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:48.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:332: TEST_recovery_pool_priority: export CEPH_ARGS 2024-04-25T07:44:48.796 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: expr 2 - 1 2024-04-25T07:44:48.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: seq 0 1 2024-04-25T07:44:48.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T07:44:48.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2024-04-25T07:44:48.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T07:44:48.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:44:48.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:44:48.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:44:48.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-04-25T07:44:48.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:48.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:44:48.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:44:48.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-04-25T07:44:48.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:44:48.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75f25b4c-c0d9-4696-bf22-4f8accbe56e9 2024-04-25T07:44:48.803 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 75f25b4c-c0d9-4696-bf22-4f8accbe56e9 2024-04-25T07:44:48.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 75f25b4c-c0d9-4696-bf22-4f8accbe56e9' 2024-04-25T07:44:48.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:44:48.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwCSpmdQYKMRAAV10tOr4q3XF84l7zsVaqng== 2024-04-25T07:44:48.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwCSpmdQYKMRAAV10tOr4q3XF84l7zsVaqng=="}' 2024-04-25T07:44:48.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75f25b4c-c0d9-4696-bf22-4f8accbe56e9 -i td/osd-recovery-prio/0/new.json 2024-04-25T07:44:49.104 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:44:49.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-04-25T07:44:49.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwCSpmdQYKMRAAV10tOr4q3XF84l7zsVaqng== --osd-uuid 75f25b4c-c0d9-4696-bf22-4f8accbe56e9 2024-04-25T07:44:49.156 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:49.155+0000 7fb2339767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:49.173 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:49.171+0000 7fb2339767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:49.174 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:49.175+0000 7fb2339767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:49.176 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:49.175+0000 7fb2339767c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T07:44:49.176 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:49.175+0000 7fb2339767c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T07:44:49.176 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:49.175+0000 7fb2339767c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T07:44:49.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:49.211+0000 7fb2339767c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-04-25T07:44:55.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-04-25T07:44:55.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:44:55.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:44:55.155 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:44:55.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:44:55.744 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:44:55.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:44:55.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:44:55.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:44:55.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:44:55.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:44:55.778 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:55.774+0000 7fb563e4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:55.781 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:55.778+0000 7fb563e4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:55.782 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:55.778+0000 7fb563e4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:56.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:44:56.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:44:56.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:44:56.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:44:56.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:44:56.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:56.202 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:44:56.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:44:56.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:56.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:56.352 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:56.350+0000 7fb563e4a7c0 -1 Falling back to public interface 2024-04-25T07:44:56.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:57.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:57.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:57.668 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:44:57.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:44:57.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:57.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:58.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:58.836 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:44:58.834+0000 7fb563e4a7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:44:59.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:59.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:59.094 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:44:59.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:44:59.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:59.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:59.612 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2209295124,v1:127.0.0.1:6803/2209295124] [v2:127.0.0.1:6804/2209295124,v1:127.0.0.1:6805/2209295124] exists,up 75f25b4c-c0d9-4696-bf22-4f8accbe56e9 2024-04-25T07:44:59.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:44:59.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:44:59.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:44:59.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T07:44:59.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2024-04-25T07:44:59.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T07:44:59.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:44:59.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:59.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:59.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:59.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:44:59.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:44:59.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-04-25T07:44:59.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:44:59.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=463ca8d2-4761-4988-bb36-9283e724766f 2024-04-25T07:44:59.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 463ca8d2-4761-4988-bb36-9283e724766f' 2024-04-25T07:44:59.620 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 463ca8d2-4761-4988-bb36-9283e724766f 2024-04-25T07:44:59.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:44:59.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7CSpmSxSDJhAApiCS+2EscOwsCMVEOyfXyw== 2024-04-25T07:44:59.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7CSpmSxSDJhAApiCS+2EscOwsCMVEOyfXyw=="}' 2024-04-25T07:44:59.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 463ca8d2-4761-4988-bb36-9283e724766f -i td/osd-recovery-prio/1/new.json 2024-04-25T07:45:00.248 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:45:00.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-04-25T07:45:00.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7CSpmSxSDJhAApiCS+2EscOwsCMVEOyfXyw== --osd-uuid 463ca8d2-4761-4988-bb36-9283e724766f 2024-04-25T07:45:00.305 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:00.302+0000 7f89d17627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:00.307 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:00.306+0000 7f89d17627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:00.309 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:00.306+0000 7f89d17627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:00.309 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:00.306+0000 7f89d17627c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T07:45:00.309 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:00.306+0000 7f89d17627c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T07:45:00.309 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:00.306+0000 7f89d17627c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T07:45:00.399 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:00.398+0000 7f89d17627c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-04-25T07:45:06.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-04-25T07:45:06.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:45:06.224 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:45:06.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:45:06.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:45:06.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:45:06.799 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:45:06.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:45:06.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:45:06.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:45:06.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:45:06.834 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:06.830+0000 7fc61d2c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:06.870 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:06.866+0000 7fc61d2c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:06.871 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:06.870+0000 7fc61d2c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:07.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:45:07.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:45:07.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:45:07.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:45:07.266 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:45:07.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:45:07.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:07.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:45:07.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:07.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:07.452 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:07.450+0000 7fc61d2c07c0 -1 Falling back to public interface 2024-04-25T07:45:07.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:08.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:08.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:08.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:45:08.718 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:45:08.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:08.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:09.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:45:10.090+0000 7fc61d2c07c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:45:10.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:10.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:10.177 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:45:10.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:45:10.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:10.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:10.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:11.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:11.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:11.648 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:45:11.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:45:11.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:11.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:45:12.065 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1768385776,v1:127.0.0.1:6811/1768385776] [v2:127.0.0.1:6812/1768385776,v1:127.0.0.1:6813/1768385776] exists,up 463ca8d2-4761-4988-bb36-9283e724766f 2024-04-25T07:45:12.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:45:12.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:45:12.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:45:12.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: seq 1 3 2024-04-25T07:45:12.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T07:45:12.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test1 1 1 2024-04-25T07:45:12.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T07:45:12.645 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test1' already exists 2024-04-25T07:45:12.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:45:13.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-04-25T07:45:14.423 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 2 2024-04-25T07:45:14.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T07:45:14.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test2 1 1 2024-04-25T07:45:14.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-04-25T07:45:15.100 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test2' already exists 2024-04-25T07:45:15.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:45:16.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-04-25T07:45:16.826 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 2 size to 2 2024-04-25T07:45:16.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T07:45:16.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test3 1 1 2024-04-25T07:45:16.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-04-25T07:45:17.513 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test3' already exists 2024-04-25T07:45:17.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:45:18.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2024-04-25T07:45:19.303 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 3 size to 2 2024-04-25T07:45:19.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:344: TEST_recovery_pool_priority: sleep 5 2024-04-25T07:45:24.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:346: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T07:45:24.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:45:24.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:45:24.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:45:24.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:45:24.332 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:45:24.332 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:45:24.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:45:24.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:45:24.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:45:24.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:45:24.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:45:24.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:45:24.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:45:24.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:45:24.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:45:24.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:45:24.919 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T07:45:24.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:45:24.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:45:24.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:45:25.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836487 2024-04-25T07:45:25.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836487 2024-04-25T07:45:25.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487' 2024-04-25T07:45:25.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:45:25.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:45:25.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672965 2024-04-25T07:45:25.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672965 2024-04-25T07:45:25.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-04-25T07:45:25.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:45:25.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836487 2024-04-25T07:45:25.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:45:25.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:45:25.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836487 2024-04-25T07:45:25.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:45:25.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836487 2024-04-25T07:45:25.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-04-25T07:45:25.264 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836487 2024-04-25T07:45:25.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:45:25.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 21474836487 2024-04-25T07:45:25.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:45:25.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672965 2024-04-25T07:45:25.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:45:25.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:45:25.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672965 2024-04-25T07:45:25.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:45:25.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672965 2024-04-25T07:45:25.674 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672965 2024-04-25T07:45:25.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-04-25T07:45:25.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:45:26.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672965 -lt 42949672965 2024-04-25T07:45:26.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:45:26.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:45:26.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:45:26.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T07:45:26.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:45:26.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:45:26.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:45:26.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:45:26.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:45:26.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:45:26.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:45:27.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T07:45:27.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:45:27.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:45:27.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:45:27.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T07:45:27.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:45:27.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:45:27.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:348: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T07:45:28.072 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:45:28.072 INFO:tasks.workunit.client.0.smithi088.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:45:19.509191+0000 0'0 26:17 [1,0] 1 [1,0] 1 0'0 2024-04-25T07:45:17.297173+0000 0'0 2024-04-25T07:45:17.297173+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:01:22.420379+0000 0 0 2024-04-25T07:45:28.072 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:45:17.299274+0000 0'0 26:27 [0,1] 0 [0,1] 0 0'0 2024-04-25T07:45:14.959806+0000 0'0 2024-04-25T07:45:14.959806+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:32:54.650952+0000 0 0 2024-04-25T07:45:28.072 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:45:15.199299+0000 0'0 26:36 [1,0] 1 [1,0] 1 0'0 2024-04-25T07:45:12.475012+0000 0'0 2024-04-25T07:45:12.475012+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:15:21.261117+0000 0 0 2024-04-25T07:45:28.072 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:45:28.072 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:45:28.072 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:45:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local PG1 2024-04-25T07:45:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local POOLNUM1 2024-04-25T07:45:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:354: TEST_recovery_pool_priority: local pool1 2024-04-25T07:45:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local chk_osd1_1 2024-04-25T07:45:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local chk_osd1_2 2024-04-25T07:45:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local PG2 2024-04-25T07:45:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local POOLNUM2 2024-04-25T07:45:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:360: TEST_recovery_pool_priority: local pool2 2024-04-25T07:45:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: local chk_osd2_1 2024-04-25T07:45:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:362: TEST_recovery_pool_priority: local chk_osd2_2 2024-04-25T07:45:28.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: seq 1 3 2024-04-25T07:45:28.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T07:45:28.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-04-25T07:45:28.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2024-04-25T07:45:28.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:45:28.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=1 2024-04-25T07:45:28.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:45:28.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=0 2024-04-25T07:45:28.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z '' ']' 2024-04-25T07:45:28.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: PG1=1.0 2024-04-25T07:45:28.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: POOLNUM1=1 2024-04-25T07:45:28.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: pool1=test1 2024-04-25T07:45:28.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:374: TEST_recovery_pool_priority: chk_osd1_1=1 2024-04-25T07:45:28.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: chk_osd1_2=0 2024-04-25T07:45:28.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T07:45:28.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-04-25T07:45:28.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2024-04-25T07:45:29.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:45:29.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=0 2024-04-25T07:45:29.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:45:29.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=1 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: PG2=2.0 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: POOLNUM2=2 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: pool2=test2 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:381: TEST_recovery_pool_priority: chk_osd2_1=0 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:382: TEST_recovery_pool_priority: chk_osd2_2=1 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: break 2024-04-25T07:45:29.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:386: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2024-04-25T07:45:29.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:388: TEST_recovery_pool_priority: '[' test2 = '' ']' 2024-04-25T07:45:29.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: seq 1 3 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:398: TEST_recovery_pool_priority: delete_pool test3 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-04-25T07:45:29.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-04-25T07:45:29.561 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test3' does not exist 2024-04-25T07:45:29.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool1_extra_prio=1 2024-04-25T07:45:29.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:403: TEST_recovery_pool_priority: pool2_extra_prio=2 2024-04-25T07:45:29.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: expr 190 + 1 2024-04-25T07:45:29.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: pool1_prio=191 2024-04-25T07:45:29.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: expr 190 + 2 2024-04-25T07:45:29.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: pool2_prio=192 2024-04-25T07:45:29.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-04-25T07:45:30.462 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 1 2024-04-25T07:45:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2024-04-25T07:45:31.241 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 recovery_priority to 1 2024-04-25T07:45:31.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:409: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2024-04-25T07:45:32.069 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 2 size to 1 2024-04-25T07:45:32.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2024-04-25T07:45:32.890 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 2 recovery_priority to 2 2024-04-25T07:45:32.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T07:45:32.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:45:32.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:45:32.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:45:32.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:45:32.910 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:45:32.910 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:45:32.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:45:32.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:45:32.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:45:33.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:45:33.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:45:33.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:45:33.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:45:33.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:45:33.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:45:33.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:45:33.449 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T07:45:33.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:45:33.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:45:33.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:45:33.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836491 2024-04-25T07:45:33.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836491 2024-04-25T07:45:33.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836491' 2024-04-25T07:45:33.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:45:33.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:45:33.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-04-25T07:45:33.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-04-25T07:45:33.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2024-04-25T07:45:33.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:45:33.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836491 2024-04-25T07:45:33.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:45:33.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:45:33.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836491 2024-04-25T07:45:33.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:45:33.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836491 2024-04-25T07:45:33.785 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836491 2024-04-25T07:45:33.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-04-25T07:45:33.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:45:34.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836489 -lt 21474836491 2024-04-25T07:45:34.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:45:35.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:45:35.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:45:35.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836491 2024-04-25T07:45:35.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:45:35.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-04-25T07:45:35.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:45:35.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-04-25T07:45:35.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:45:35.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-04-25T07:45:35.636 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672969 2024-04-25T07:45:35.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-04-25T07:45:35.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:45:36.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-04-25T07:45:36.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:45:36.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:45:36.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:45:36.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T07:45:36.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:45:36.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:45:36.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:45:36.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:45:36.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:45:36.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:45:36.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:45:36.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T07:45:36.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:45:36.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:45:36.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:45:37.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T07:45:37.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:45:37.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:45:37.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:413: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-04-25T07:45:37.570 INFO:tasks.workunit.client.0.smithi088.stderr:10+0 records in 2024-04-25T07:45:37.570 INFO:tasks.workunit.client.0.smithi088.stderr:10+0 records out 2024-04-25T07:45:37.571 INFO:tasks.workunit.client.0.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0411739 s, 255 MB/s 2024-04-25T07:45:37.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: p=1 2024-04-25T07:45:37.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-04-25T07:45:37.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-04-25T07:45:37.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:37.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-04-25T07:45:38.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:38.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-04-25T07:45:38.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:38.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-04-25T07:45:38.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:38.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-04-25T07:45:39.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:39.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-04-25T07:45:39.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:39.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-04-25T07:45:39.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:39.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-04-25T07:45:40.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:40.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-04-25T07:45:40.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:40.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-04-25T07:45:40.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:40.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-04-25T07:45:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:41.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-04-25T07:45:41.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:41.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-04-25T07:45:42.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:42.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-04-25T07:45:42.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:42.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-04-25T07:45:42.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:42.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-04-25T07:45:43.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:43.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-04-25T07:45:43.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:43.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-04-25T07:45:43.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:43.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-04-25T07:45:44.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:44.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-04-25T07:45:44.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:44.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-04-25T07:45:45.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:45.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-04-25T07:45:45.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:45.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-04-25T07:45:45.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:45.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-04-25T07:45:46.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:46.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-04-25T07:45:46.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:46.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-04-25T07:45:46.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:46.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-04-25T07:45:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-04-25T07:45:47.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:47.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-04-25T07:45:48.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:48.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-04-25T07:45:48.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:48.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-04-25T07:45:48.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:48.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-04-25T07:45:49.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:49.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-04-25T07:45:49.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:49.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-04-25T07:45:50.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:50.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-04-25T07:45:50.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:50.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-04-25T07:45:50.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:50.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-04-25T07:45:51.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:51.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-04-25T07:45:51.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:51.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-04-25T07:45:52.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:52.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-04-25T07:45:52.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:52.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-04-25T07:45:52.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:52.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-04-25T07:45:53.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:53.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-04-25T07:45:53.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:53.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-04-25T07:45:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-04-25T07:45:54.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:54.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-04-25T07:45:54.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:54.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-04-25T07:45:55.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:55.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-04-25T07:45:55.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:55.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-04-25T07:45:55.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:55.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-04-25T07:45:56.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:56.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-04-25T07:45:56.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:56.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-04-25T07:45:56.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:56.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-04-25T07:45:57.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:57.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-04-25T07:45:57.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:57.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-04-25T07:45:58.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:58.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-04-25T07:45:58.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:58.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-04-25T07:45:58.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:58.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-04-25T07:45:59.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:59.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-04-25T07:45:59.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:45:59.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-04-25T07:46:00.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:00.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-04-25T07:46:00.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:00.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-04-25T07:46:00.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:00.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-04-25T07:46:01.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:01.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-04-25T07:46:01.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:01.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-04-25T07:46:01.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:01.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-04-25T07:46:02.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:02.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-04-25T07:46:02.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:02.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-04-25T07:46:02.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:02.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-04-25T07:46:03.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:03.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-04-25T07:46:03.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:03.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-04-25T07:46:04.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:04.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-04-25T07:46:04.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:04.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-04-25T07:46:04.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:04.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-04-25T07:46:05.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:05.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-04-25T07:46:05.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:05.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-04-25T07:46:06.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:06.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-04-25T07:46:06.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:06.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-04-25T07:46:06.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:06.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-04-25T07:46:07.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:07.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-04-25T07:46:07.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:07.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-04-25T07:46:07.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:07.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-04-25T07:46:08.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:08.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-04-25T07:46:08.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:08.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-04-25T07:46:09.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:09.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-04-25T07:46:09.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:09.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-04-25T07:46:09.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:09.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-04-25T07:46:10.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:10.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-04-25T07:46:10.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:10.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-04-25T07:46:10.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:10.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-04-25T07:46:11.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:11.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-04-25T07:46:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-04-25T07:46:12.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:12.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-04-25T07:46:12.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:12.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-04-25T07:46:12.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:12.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-04-25T07:46:13.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:13.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-04-25T07:46:13.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:13.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-04-25T07:46:13.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:13.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-04-25T07:46:14.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:14.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-04-25T07:46:14.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:14.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-04-25T07:46:14.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:14.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-04-25T07:46:15.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:15.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-04-25T07:46:15.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:15.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-04-25T07:46:16.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:16.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-04-25T07:46:16.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:16.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-04-25T07:46:16.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:16.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-04-25T07:46:17.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:17.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-04-25T07:46:17.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:17.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-04-25T07:46:17.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:17.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-04-25T07:46:18.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:18.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-04-25T07:46:18.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:18.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-04-25T07:46:18.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:18.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-04-25T07:46:19.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:19.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-04-25T07:46:19.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:19.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-04-25T07:46:20.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:20.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-04-25T07:46:20.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:20.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-04-25T07:46:20.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:20.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-04-25T07:46:21.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:21.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-04-25T07:46:21.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:21.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-04-25T07:46:21.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:21.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-04-25T07:46:22.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:22.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-04-25T07:46:22.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:22.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-04-25T07:46:22.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:22.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-04-25T07:46:23.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:23.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-04-25T07:46:23.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:23.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-04-25T07:46:24.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:24.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-04-25T07:46:24.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:24.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-04-25T07:46:24.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:24.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-04-25T07:46:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:25.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-04-25T07:46:25.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:25.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-04-25T07:46:26.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:26.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-04-25T07:46:26.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:26.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-04-25T07:46:26.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:26.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-04-25T07:46:27.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:27.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-04-25T07:46:27.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:27.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-04-25T07:46:27.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:27.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-04-25T07:46:28.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:28.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-04-25T07:46:28.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:28.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-04-25T07:46:28.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:28.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-04-25T07:46:29.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:29.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-04-25T07:46:29.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:29.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-04-25T07:46:30.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:30.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-04-25T07:46:30.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:30.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-04-25T07:46:30.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:30.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-04-25T07:46:31.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:31.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-04-25T07:46:31.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:31.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-04-25T07:46:32.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:32.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-04-25T07:46:32.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:32.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-04-25T07:46:32.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:32.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-04-25T07:46:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-04-25T07:46:33.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:33.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-04-25T07:46:33.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:33.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-04-25T07:46:34.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:34.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-04-25T07:46:34.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:34.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-04-25T07:46:34.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:34.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-04-25T07:46:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-04-25T07:46:35.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:35.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-04-25T07:46:36.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:36.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-04-25T07:46:36.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:36.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-04-25T07:46:36.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:36.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-04-25T07:46:37.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:37.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-04-25T07:46:37.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:37.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-04-25T07:46:38.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:38.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-04-25T07:46:38.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:38.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-04-25T07:46:38.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:38.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-04-25T07:46:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-04-25T07:46:39.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:39.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-04-25T07:46:39.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:39.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-04-25T07:46:40.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:40.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-04-25T07:46:40.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:40.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-04-25T07:46:41.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:41.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-04-25T07:46:41.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:41.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-04-25T07:46:41.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:41.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-04-25T07:46:42.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:42.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-04-25T07:46:42.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:42.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-04-25T07:46:42.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:42.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-04-25T07:46:43.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:43.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-04-25T07:46:43.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:43.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-04-25T07:46:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-04-25T07:46:44.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:44.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-04-25T07:46:44.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:44.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-04-25T07:46:45.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:45.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-04-25T07:46:45.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:45.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-04-25T07:46:45.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:45.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-04-25T07:46:46.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:46.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-04-25T07:46:46.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:46.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-04-25T07:46:46.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:46.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-04-25T07:46:47.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:47.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-04-25T07:46:47.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:47.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-04-25T07:46:48.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:48.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-04-25T07:46:48.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:48.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-04-25T07:46:48.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:48.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-04-25T07:46:49.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:49.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-04-25T07:46:49.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:49.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-04-25T07:46:49.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:49.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-04-25T07:46:50.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:50.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-04-25T07:46:50.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:50.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-04-25T07:46:51.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:51.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-04-25T07:46:51.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:51.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-04-25T07:46:51.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:51.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-04-25T07:46:52.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:52.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-04-25T07:46:52.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 1 + 1 2024-04-25T07:46:52.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=2 2024-04-25T07:46:52.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-04-25T07:46:52.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-04-25T07:46:52.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:52.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2024-04-25T07:46:52.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:52.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2024-04-25T07:46:53.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:53.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2024-04-25T07:46:53.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:53.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2024-04-25T07:46:54.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:54.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2024-04-25T07:46:54.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:54.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2024-04-25T07:46:54.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:54.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2024-04-25T07:46:55.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:55.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2024-04-25T07:46:55.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:55.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2024-04-25T07:46:56.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:56.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2024-04-25T07:46:56.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:56.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2024-04-25T07:46:56.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2024-04-25T07:46:57.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:57.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2024-04-25T07:46:57.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:57.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2024-04-25T07:46:58.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:58.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2024-04-25T07:46:58.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:58.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2024-04-25T07:46:58.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:58.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2024-04-25T07:46:59.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:59.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2024-04-25T07:46:59.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:59.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2024-04-25T07:46:59.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:46:59.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2024-04-25T07:47:00.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:00.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2024-04-25T07:47:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2024-04-25T07:47:01.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:01.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2024-04-25T07:47:01.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:01.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2024-04-25T07:47:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2024-04-25T07:47:02.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:02.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2024-04-25T07:47:02.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:02.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2024-04-25T07:47:02.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:02.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2024-04-25T07:47:03.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:03.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2024-04-25T07:47:03.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:03.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2024-04-25T07:47:04.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:04.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2024-04-25T07:47:04.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:04.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2024-04-25T07:47:04.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:04.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2024-04-25T07:47:05.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:05.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2024-04-25T07:47:05.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:05.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2024-04-25T07:47:05.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:05.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2024-04-25T07:47:06.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:06.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2024-04-25T07:47:06.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:06.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2024-04-25T07:47:07.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:07.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2024-04-25T07:47:07.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:07.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2024-04-25T07:47:07.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:07.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2024-04-25T07:47:08.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:08.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2024-04-25T07:47:08.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:08.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2024-04-25T07:47:08.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:08.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2024-04-25T07:47:09.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:09.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2024-04-25T07:47:09.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:09.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2024-04-25T07:47:10.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:10.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2024-04-25T07:47:10.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:10.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2024-04-25T07:47:10.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:10.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2024-04-25T07:47:11.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:11.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2024-04-25T07:47:11.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:11.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2024-04-25T07:47:11.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:11.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2024-04-25T07:47:12.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:12.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2024-04-25T07:47:12.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:12.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2024-04-25T07:47:13.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:13.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2024-04-25T07:47:13.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:13.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2024-04-25T07:47:13.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:13.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2024-04-25T07:47:14.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:14.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2024-04-25T07:47:14.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:14.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2024-04-25T07:47:14.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:14.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2024-04-25T07:47:15.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:15.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2024-04-25T07:47:15.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:15.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2024-04-25T07:47:16.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:16.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2024-04-25T07:47:16.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:16.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2024-04-25T07:47:16.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:16.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2024-04-25T07:47:17.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:17.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2024-04-25T07:47:17.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:17.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2024-04-25T07:47:17.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:17.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2024-04-25T07:47:18.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:18.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2024-04-25T07:47:18.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:18.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2024-04-25T07:47:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:19.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2024-04-25T07:47:19.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:19.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2024-04-25T07:47:19.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:19.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2024-04-25T07:47:20.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:20.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2024-04-25T07:47:20.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:20.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2024-04-25T07:47:20.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:20.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2024-04-25T07:47:21.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:21.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2024-04-25T07:47:21.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:21.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2024-04-25T07:47:22.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:22.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2024-04-25T07:47:22.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:22.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2024-04-25T07:47:22.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:22.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2024-04-25T07:47:23.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:23.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2024-04-25T07:47:23.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:23.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2024-04-25T07:47:24.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:24.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2024-04-25T07:47:24.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:24.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2024-04-25T07:47:24.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:24.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2024-04-25T07:47:25.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:25.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2024-04-25T07:47:25.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:25.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2024-04-25T07:47:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2024-04-25T07:47:26.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:26.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2024-04-25T07:47:26.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:26.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2024-04-25T07:47:27.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:27.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2024-04-25T07:47:27.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:27.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2024-04-25T07:47:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2024-04-25T07:47:28.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:28.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2024-04-25T07:47:28.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:28.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2024-04-25T07:47:29.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:29.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2024-04-25T07:47:29.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:29.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2024-04-25T07:47:30.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:30.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2024-04-25T07:47:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2024-04-25T07:47:30.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:30.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2024-04-25T07:47:31.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:31.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2024-04-25T07:47:31.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:31.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2024-04-25T07:47:32.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:32.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2024-04-25T07:47:32.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:32.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2024-04-25T07:47:32.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:32.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2024-04-25T07:47:33.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:33.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2024-04-25T07:47:33.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:33.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2024-04-25T07:47:33.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2024-04-25T07:47:34.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:34.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2024-04-25T07:47:34.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:34.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2024-04-25T07:47:35.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:35.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2024-04-25T07:47:35.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:35.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2024-04-25T07:47:35.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:35.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2024-04-25T07:47:36.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:36.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2024-04-25T07:47:36.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:36.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2024-04-25T07:47:36.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:36.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2024-04-25T07:47:37.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:37.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2024-04-25T07:47:37.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:37.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2024-04-25T07:47:38.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:38.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2024-04-25T07:47:38.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:38.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2024-04-25T07:47:38.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:38.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2024-04-25T07:47:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2024-04-25T07:47:39.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:39.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2024-04-25T07:47:40.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:40.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2024-04-25T07:47:40.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:40.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2024-04-25T07:47:40.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:40.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2024-04-25T07:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2024-04-25T07:47:41.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:41.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2024-04-25T07:47:41.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:41.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2024-04-25T07:47:42.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:42.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2024-04-25T07:47:42.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:42.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2024-04-25T07:47:43.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:43.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2024-04-25T07:47:43.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:43.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2024-04-25T07:47:43.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:43.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2024-04-25T07:47:44.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:44.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2024-04-25T07:47:44.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2024-04-25T07:47:45.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:45.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2024-04-25T07:47:45.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:45.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2024-04-25T07:47:45.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:45.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2024-04-25T07:47:46.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:46.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2024-04-25T07:47:46.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:46.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2024-04-25T07:47:46.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:46.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2024-04-25T07:47:47.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:47.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2024-04-25T07:47:47.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:47.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2024-04-25T07:47:48.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:48.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2024-04-25T07:47:48.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:48.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2024-04-25T07:47:48.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:48.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2024-04-25T07:47:49.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:49.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2024-04-25T07:47:49.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:49.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2024-04-25T07:47:49.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:49.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2024-04-25T07:47:50.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:50.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2024-04-25T07:47:50.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:50.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2024-04-25T07:47:51.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:51.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2024-04-25T07:47:51.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:51.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2024-04-25T07:47:51.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:51.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2024-04-25T07:47:52.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:52.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2024-04-25T07:47:52.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:52.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2024-04-25T07:47:52.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:52.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2024-04-25T07:47:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2024-04-25T07:47:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2024-04-25T07:47:54.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:54.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2024-04-25T07:47:54.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:54.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2024-04-25T07:47:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2024-04-25T07:47:55.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:55.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2024-04-25T07:47:55.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:55.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2024-04-25T07:47:56.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:56.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2024-04-25T07:47:56.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:56.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2024-04-25T07:47:56.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:56.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2024-04-25T07:47:57.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:57.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2024-04-25T07:47:57.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:57.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2024-04-25T07:47:58.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:58.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2024-04-25T07:47:58.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:58.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2024-04-25T07:47:58.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:58.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2024-04-25T07:47:59.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:59.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2024-04-25T07:47:59.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:47:59.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2024-04-25T07:48:00.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:00.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2024-04-25T07:48:00.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:00.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2024-04-25T07:48:00.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:00.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2024-04-25T07:48:01.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:01.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2024-04-25T07:48:01.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:01.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2024-04-25T07:48:02.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:02.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2024-04-25T07:48:02.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:02.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2024-04-25T07:48:02.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:02.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2024-04-25T07:48:03.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:03.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2024-04-25T07:48:03.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:03.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2024-04-25T07:48:04.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:04.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2024-04-25T07:48:04.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:04.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2024-04-25T07:48:04.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:04.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2024-04-25T07:48:05.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:05.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2024-04-25T07:48:05.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:05.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2024-04-25T07:48:05.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:05.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2024-04-25T07:48:06.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:06.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2024-04-25T07:48:06.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:06.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2024-04-25T07:48:07.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:07.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2024-04-25T07:48:07.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:07.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2024-04-25T07:48:07.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:07.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2024-04-25T07:48:08.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:08.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2024-04-25T07:48:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2024-04-25T07:48:08.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T07:48:08.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2024-04-25T07:48:09.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 2 + 1 2024-04-25T07:48:09.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=3 2024-04-25T07:48:09.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2024-04-25T07:48:09.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-04-25T07:48:09.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-04-25T07:48:09.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-04-25T07:48:09.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-04-25T07:48:09.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-04-25T07:48:09.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T07:48:09.360 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T07:48:09.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T07:48:09.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T07:48:09.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T07:48:10.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: local otherosd=null 2024-04-25T07:48:10.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T07:48:10.589 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:48:10.589 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 189 0 0 0 0 1981808640 0 0 567 0 567 active+clean 2024-04-25T07:45:32.382486+0000 35'567 35:618 [0] 0 [0] 0 0'0 2024-04-25T07:45:14.959806+0000 0'0 2024-04-25T07:45:14.959806+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:30:55.517938+0000 0 0 2024-04-25T07:48:10.590 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T07:45:30.708696+0000 35'600 35:658 [1] 1 [1] 1 0'0 2024-04-25T07:45:12.475012+0000 0'0 2024-04-25T07:45:12.475012+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:46:47.068421+0000 0 0 2024-04-25T07:48:10.590 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:48:10.590 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:48:10.590 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:48:10.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ERRORS=0 2024-04-25T07:48:10.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-04-25T07:48:11.266 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 2 2024-04-25T07:48:11.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-04-25T07:48:12.571 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 2 size to 2 2024-04-25T07:48:12.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T07:48:13.007 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:48:13.008 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T07:45:32.382486+0000 35'600 35:651 [0] 0 [0] 0 0'0 2024-04-25T07:45:14.959806+0000 0'0 2024-04-25T07:45:14.959806+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:30:55.517938+0000 0 0 2024-04-25T07:48:13.008 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 peering 2024-04-25T07:48:11.463209+0000 35'600 35:659 [1,0] 1 [1,0] 1 0'0 2024-04-25T07:45:12.475012+0000 0'0 2024-04-25T07:45:12.475012+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:48:13.008 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:48:13.008 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:48:13.008 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:48:13.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:436: TEST_recovery_pool_priority: set -o pipefail 2024-04-25T07:48:13.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: count=0 2024-04-25T07:48:13.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T07:48:13.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T07:48:13.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T07:48:13.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T07:48:13.428 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:48:13.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-04-25T07:48:13.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-04-25T07:48:15.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 0 -eq 10 2024-04-25T07:48:15.447 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 0 + 1 2024-04-25T07:48:15.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=1 2024-04-25T07:48:15.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T07:48:15.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T07:48:15.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T07:48:15.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T07:48:15.853 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:48:15.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 1 == 2 2024-04-25T07:48:15.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-04-25T07:48:17.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 1 -eq 10 2024-04-25T07:48:17.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 1 + 1 2024-04-25T07:48:17.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=2 2024-04-25T07:48:17.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T07:48:17.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T07:48:17.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T07:48:17.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T07:48:18.284 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:48:18.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 2 == 2 2024-04-25T07:48:18.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: break 2024-04-25T07:48:18.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: set +o pipefail 2024-04-25T07:48:18.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T07:48:18.682 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:48:18.683 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 200 0 199 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-04-25T07:48:14.062497+0000 35'600 42:669 [0,1] 0 [0] 0 0'0 2024-04-25T07:45:14.959806+0000 0'0 2024-04-25T07:45:14.959806+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:48:18.683 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 200 0 195 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-04-25T07:48:12.717426+0000 35'600 42:683 [1,0] 1 [1] 1 0'0 2024-04-25T07:45:12.475012+0000 0'0 2024-04-25T07:45:12.475012+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:48:18.683 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:48:18.683 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:48:18.683 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:48:18.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.1 2024-04-25T07:48:18.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:48:18.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:48:18.699 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:48:18.699 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:48:18.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:48:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2024-04-25T07:48:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:48:18.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.1 2024-04-25T07:48:18.807 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 2024-04-25T07:48:18.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T07:48:18.808 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:48:18.808 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:48:18.808 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:48:18.808 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:48:18.808 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:48:18.808 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:48:18.808 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:48:18.808 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 191, 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "2.0", 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 192, 2024-04-25T07:48:18.809 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:48:18.810 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:48:18.810 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:48:18.810 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:48:18.810 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: get_asok_path osd.0 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.0.asok 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: CEPH_ARGS= 2024-04-25T07:48:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.0.asok dump_recovery_reservations 2024-04-25T07:48:18.931 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 2024-04-25T07:48:18.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:460: TEST_recovery_pool_priority: echo osd.0 2024-04-25T07:48:18.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T07:48:18.932 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:48:18.932 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "2.0", 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 192, 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:48:18.933 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 191, 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:48:18.934 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:48:18.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T07:48:18.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T07:48:18.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-04-25T07:48:18.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: ITEM=1.0 2024-04-25T07:48:18.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:465: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-04-25T07:48:18.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T07:48:18.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T07:48:19.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: PRIO=191 2024-04-25T07:48:19.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:471: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-04-25T07:48:19.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-04-25T07:48:19.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T07:48:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-04-25T07:48:19.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: ITEM=1.0 2024-04-25T07:48:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:480: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-04-25T07:48:19.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T07:48:19.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-04-25T07:48:19.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: PRIO=191 2024-04-25T07:48:19.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:486: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-04-25T07:48:19.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T07:48:19.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T07:48:19.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-04-25T07:48:19.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: ITEM=2.0 2024-04-25T07:48:19.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:495: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-04-25T07:48:19.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T07:48:19.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T07:48:19.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: PRIO=192 2024-04-25T07:48:19.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:501: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-04-25T07:48:19.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T07:48:19.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-04-25T07:48:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-04-25T07:48:19.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: ITEM=2.0 2024-04-25T07:48:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:510: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-04-25T07:48:19.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T07:48:19.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-04-25T07:48:19.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: PRIO=192 2024-04-25T07:48:19.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:516: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-04-25T07:48:19.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:523: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T07:48:19.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:48:19.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:48:19.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:48:19.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:48:19.232 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:48:19.232 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:48:19.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:48:19.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:48:19.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:48:19.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:48:19.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:48:19.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:48:19.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:48:19.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:48:19.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:48:19.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:48:19.795 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T07:48:19.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:48:19.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:48:19.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:48:20.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836526 2024-04-25T07:48:20.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836526 2024-04-25T07:48:20.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526' 2024-04-25T07:48:20.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:48:20.089 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:48:20.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673004 2024-04-25T07:48:20.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673004 2024-04-25T07:48:20.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673004' 2024-04-25T07:48:20.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:48:20.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836526 2024-04-25T07:48:20.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:48:20.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:48:20.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836526 2024-04-25T07:48:20.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:48:20.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836526 2024-04-25T07:48:20.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-04-25T07:48:20.258 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836526 2024-04-25T07:48:20.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:48:20.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836524 -lt 21474836526 2024-04-25T07:48:20.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:48:21.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:48:21.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:48:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836526 2024-04-25T07:48:22.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:48:22.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673004 2024-04-25T07:48:22.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:48:22.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:48:22.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673004 2024-04-25T07:48:22.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:48:22.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673004 2024-04-25T07:48:22.246 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673004 2024-04-25T07:48:22.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-04-25T07:48:22.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:48:22.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673004 -lt 42949673004 2024-04-25T07:48:22.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:48:22.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:22.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:23.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T07:48:23.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:23.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:23.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:23.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:23.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:23.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:23.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:24.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:24.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:24.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:24.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:24.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:48:24.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:48:24.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:48:24.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:24.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:24.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:24.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:24.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:24.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:24.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:24.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:24.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:24.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:25.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:25.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:25.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:25.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:25.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:25.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:25.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:26.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586385 2024-04-25T07:48:26.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586385 '!=' null 2024-04-25T07:48:26.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:26.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:26.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:26.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:26.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:26.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:26.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:26.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:26.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:26.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:26.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:26.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:26.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:26.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:26.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:27.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:27.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:27.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:28.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28826928 2024-04-25T07:48:28.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28826928 '!=' null 2024-04-25T07:48:28.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:28.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:28.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:28.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:28.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:28.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:28.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:28.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:28.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:28.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:28.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:29.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:29.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:29.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:29.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:29.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:29.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:29.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:29.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:30.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459014 2024-04-25T07:48:30.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459014 '!=' null 2024-04-25T07:48:30.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:30.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:30.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:30.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:30.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:30.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:30.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:30.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:30.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:31.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:31.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:31.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:31.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:31.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:31.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:31.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:31.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:31.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:32.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573699 2024-04-25T07:48:32.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573699 '!=' null 2024-04-25T07:48:32.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:32.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:32.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:32.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:32.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:32.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:32.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:32.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:32.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:32.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:32.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:32.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:32.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:32.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:32.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:33.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:33.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:33.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:33.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:33.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:33.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:33.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:33.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:33.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:34.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25332138 2024-04-25T07:48:34.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25332138 '!=' null 2024-04-25T07:48:34.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:34.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:34.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:34.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:34.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:34.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:34.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:34.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:34.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:34.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:34.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:34.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:34.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:34.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:34.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:35.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:35.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:35.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:35.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:35.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:35.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:35.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:35.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:35.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:36.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573029 2024-04-25T07:48:36.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573029 '!=' null 2024-04-25T07:48:36.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:36.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:36.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:36.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:36.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:36.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:36.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:36.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:36.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:36.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:36.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:36.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:36.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:36.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:36.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:37.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:37.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:37.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:37.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:37.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:37.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:37.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:37.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:37.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:38.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573343 2024-04-25T07:48:38.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573343 '!=' null 2024-04-25T07:48:38.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:38.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:38.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:38.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:38.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:38.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:38.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:38.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:38.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:38.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:38.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:38.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:38.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:38.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:38.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:39.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:39.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:39.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:39.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573343 2024-04-25T07:48:39.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573343 '!=' null 2024-04-25T07:48:39.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:39.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:39.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:39.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:39.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:39.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:39.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:39.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:39.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:39.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:39.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:40.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:40.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:40.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:40.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:40.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:40.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:40.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:41.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24458974 2024-04-25T07:48:41.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24458974 '!=' null 2024-04-25T07:48:41.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:41.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:41.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:41.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:41.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:41.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:41.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:41.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:41.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:41.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:41.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:42.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:42.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:42.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:42.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:42.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:42.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:42.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:42.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:42.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:42.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:42.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:42.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:43.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574130 2024-04-25T07:48:43.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574130 '!=' null 2024-04-25T07:48:43.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:43.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:43.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:43.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:43.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:43.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:43.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:43.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:43.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:43.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:43.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:44.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:44.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:44.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:44.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:44.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:44.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:44.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:45.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574595 2024-04-25T07:48:45.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574595 '!=' null 2024-04-25T07:48:45.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:45.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:45.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:45.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:45.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:45.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:45.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:45.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:45.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:45.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:45.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:45.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:45.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:45.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:45.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:46.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:46.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:46.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701631 2024-04-25T07:48:46.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701631 '!=' null 2024-04-25T07:48:46.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:46.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:46.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:47.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:47.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:47.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:47.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:47.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:47.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:47.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:47.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:47.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:47.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:47.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:47.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:48.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:48.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:48.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:48.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:48.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:48.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828222 2024-04-25T07:48:48.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828222 '!=' null 2024-04-25T07:48:48.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:48.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:48.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:48.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:48.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:48.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:48.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:48.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:48.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:48.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:48.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:49.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:49.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:49.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:49.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:50.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:50.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:50.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:50.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:50.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:50.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:50.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:50.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:50.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713001 2024-04-25T07:48:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713001 '!=' null 2024-04-25T07:48:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:50.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:50.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:50.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:50.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:50.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:50.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:50.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:50.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:50.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:51.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:51.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:51.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:51.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:51.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:51.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:51.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:51.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:51.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:51.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:51.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:51.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:51.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:52.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828737 2024-04-25T07:48:52.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828737 '!=' null 2024-04-25T07:48:52.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:52.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:52.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:52.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:52.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:52.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:52.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:52.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:52.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:52.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:52.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:52.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:52.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:52.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:52.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:53.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:53.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:53.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:53.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:53.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:53.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:53.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:53.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:53.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:54.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28829369 2024-04-25T07:48:54.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28829369 '!=' null 2024-04-25T07:48:54.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:54.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:54.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:54.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:54.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:54.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:54.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:54.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:54.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:54.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:54.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:54.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:54.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:54.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:54.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:55.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:55.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:55.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:55.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:55.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:55.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:55.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:55.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:55.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:56.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28829000 2024-04-25T07:48:56.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28829000 '!=' null 2024-04-25T07:48:56.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:56.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:56.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:56.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:56.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:56.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:56.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:56.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:56.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:56.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:56.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:56.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:56.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:56.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:56.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:57.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:57.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:57.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:57.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:57.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:57.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:57.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:57.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:57.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:58.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29702049 2024-04-25T07:48:58.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29702049 '!=' null 2024-04-25T07:48:58.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:58.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:58.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:58.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:58.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:58.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:58.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:58.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:58.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:58.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:48:58.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:58.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:48:58.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:48:58.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:48:58.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:48:59.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:48:59.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:48:59.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:48:59.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:48:59.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:48:59.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:48:59.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:48:59.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:48:59.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:48:59.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460712 2024-04-25T07:48:59.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460712 '!=' null 2024-04-25T07:48:59.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:48:59.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:48:59.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:48:59.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:48:59.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:48:59.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:48:59.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:48:59.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:48:59.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:48:59.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:48:59.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:00.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:00.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:00.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:00.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:00.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:00.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:00.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:00.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:00.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:00.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:00.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:00.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:00.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:01.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460712 2024-04-25T07:49:01.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460712 '!=' null 2024-04-25T07:49:01.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:01.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:01.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:01.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:01.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:01.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:01.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:01.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:01.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:01.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:01.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:01.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:01.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:01.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:01.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:02.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:02.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:02.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:02.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:02.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:02.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:02.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:02.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:02.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:03.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30576568 2024-04-25T07:49:03.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30576568 '!=' null 2024-04-25T07:49:03.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:03.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:03.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:03.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:03.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:03.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:03.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:03.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:03.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:03.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:03.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:04.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:04.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:04.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:04.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:04.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:04.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:04.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:04.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:04.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:04.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:04.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:04.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:04.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:05.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29702623 2024-04-25T07:49:05.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29702623 '!=' null 2024-04-25T07:49:05.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:05.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:05.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:05.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:05.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:05.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:05.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:05.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:05.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:05.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:05.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:06.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:06.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:06.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:06.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:06.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:06.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:06.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:06.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:07.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828516 2024-04-25T07:49:07.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828516 '!=' null 2024-04-25T07:49:07.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:07.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:07.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:07.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:07.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:07.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:07.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:07.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:07.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:07.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:07.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:07.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:07.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:07.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:07.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:08.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:08.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:08.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:08.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:08.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:08.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:08.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:08.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:08.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828791 2024-04-25T07:49:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828791 '!=' null 2024-04-25T07:49:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:09.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:09.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:09.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:09.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:09.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:09.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:09.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:09.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:10.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:10.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:10.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:10.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:10.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:10.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:10.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:10.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:10.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:10.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713532 2024-04-25T07:49:10.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713532 '!=' null 2024-04-25T07:49:10.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:10.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:10.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:10.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:10.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:10.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:10.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:10.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:10.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:10.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:10.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:11.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:11.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:11.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:11.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:11.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:11.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:11.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:11.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:11.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:11.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:11.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:11.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:11.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828653 2024-04-25T07:49:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828653 '!=' null 2024-04-25T07:49:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:12.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:12.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:12.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:12.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:12.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:12.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:12.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:12.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:12.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:12.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:12.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:12.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:13.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:13.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:13.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:14.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828378 2024-04-25T07:49:14.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828378 '!=' null 2024-04-25T07:49:14.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:14.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:14.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:14.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:14.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:14.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:14.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:14.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:14.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:14.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:14.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:14.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:14.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:14.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:14.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:15.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:15.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:15.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:15.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:15.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:15.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:15.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:15.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:15.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:16.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701483 2024-04-25T07:49:16.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701483 '!=' null 2024-04-25T07:49:16.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:16.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:16.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:16.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:16.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:16.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:16.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:16.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:16.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:16.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:16.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:16.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:16.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:16.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:16.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:17.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:17.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:17.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:17.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:17.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:17.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:17.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:17.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:17.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:18.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574744 2024-04-25T07:49:18.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574744 '!=' null 2024-04-25T07:49:18.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:18.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:18.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:18.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:18.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:18.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:18.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:18.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:18.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:18.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:18.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:18.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:18.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:18.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:18.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:19.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:19.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:19.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459457 2024-04-25T07:49:19.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459457 '!=' null 2024-04-25T07:49:19.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:19.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:19.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:20.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:20.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:20.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:20.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:20.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:20.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:20.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:20.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:20.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:20.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:20.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:20.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:21.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:21.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:21.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:21.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:21.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:21.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:21.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:21.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:21.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:21.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459457 2024-04-25T07:49:21.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459457 '!=' null 2024-04-25T07:49:21.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:21.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:21.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:21.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:21.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:21.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:21.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:21.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:21.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:21.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:21.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:22.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:22.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:22.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:22.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:22.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:22.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:22.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:22.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:22.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:22.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:22.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:22.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:22.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:23.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573779 2024-04-25T07:49:23.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573779 '!=' null 2024-04-25T07:49:23.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:23.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:23.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:23.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:23.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:23.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:23.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:23.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:23.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:23.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:23.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:24.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:24.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:24.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:24.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:24.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:24.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:24.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:24.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:24.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:24.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:24.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:24.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:24.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:25.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573779 2024-04-25T07:49:25.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573779 '!=' null 2024-04-25T07:49:25.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:25.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:25.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:25.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:25.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:25.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:25.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:25.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:25.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:25.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:25.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:25.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:25.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:25.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:26.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:26.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:26.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:26.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:26.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:26.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:26.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:26.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:26.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:27.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573171 2024-04-25T07:49:27.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573171 '!=' null 2024-04-25T07:49:27.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:27.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:27.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:27.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:27.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:27.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:27.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:27.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:27.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:27.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:27.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:28.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:28.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:28.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:28.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:28.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:28.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:28.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:28.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:28.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:28.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:28.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:28.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:28.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:29.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573306 2024-04-25T07:49:29.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573306 '!=' null 2024-04-25T07:49:29.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:29.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:29.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:29.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:29.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:29.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:29.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:29.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:29.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:29.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:29.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:29.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:29.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:29.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:29.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:30.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:30.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:31.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459065 2024-04-25T07:49:31.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459065 '!=' null 2024-04-25T07:49:31.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:31.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:31.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:31.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:31.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:31.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:31.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:31.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:31.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:31.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:31.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:31.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:31.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:31.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:31.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:32.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:32.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:32.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:32.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:32.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:32.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:32.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:32.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:32.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:32.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29700291 2024-04-25T07:49:32.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29700291 '!=' null 2024-04-25T07:49:32.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:32.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:32.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:32.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:32.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:32.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:32.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:32.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:32.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:32.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:32.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:33.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:33.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:33.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:33.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:33.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:33.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:33.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:33.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:33.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:33.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:33.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:33.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:33.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:34.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28826797 2024-04-25T07:49:34.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28826797 '!=' null 2024-04-25T07:49:34.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:34.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:34.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:34.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:34.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:34.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:34.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:34.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:34.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:34.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:34.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:34.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:34.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:34.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:34.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:35.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:35.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:35.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:36.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827208 2024-04-25T07:49:36.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827208 '!=' null 2024-04-25T07:49:36.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:36.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:36.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:36.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:36.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:36.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:36.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:36.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:36.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:36.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:36.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:36.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:36.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:36.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:36.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:37.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:37.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:37.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:37.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:37.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:37.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:37.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:37.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:37.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:38.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827519 2024-04-25T07:49:38.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827519 '!=' null 2024-04-25T07:49:38.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:38.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:38.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:38.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:38.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:38.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:38.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:38.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:38.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:38.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:38.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:38.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:38.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:38.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:38.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:39.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:39.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:39.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:40.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712293 2024-04-25T07:49:40.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712293 '!=' null 2024-04-25T07:49:40.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:40.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:40.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:40.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:40.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:40.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:40.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:40.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:40.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:40.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:40.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:40.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:40.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:40.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:40.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:41.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:41.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:42.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28826872 2024-04-25T07:49:42.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28826872 '!=' null 2024-04-25T07:49:42.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:42.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:42.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:42.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:42.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:42.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:42.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:42.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:42.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:42.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:42.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:42.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:42.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:42.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:42.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:43.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:43.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:43.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:43.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:43.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:43.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:43.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:43.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:43.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:44.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29700394 2024-04-25T07:49:44.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29700394 '!=' null 2024-04-25T07:49:44.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:44.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:44.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:44.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:44.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:44.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:44.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:44.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:44.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:44.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:44.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:45.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:45.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:45.051 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:45.051 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:45.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:45.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:46.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573631 2024-04-25T07:49:46.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573631 '!=' null 2024-04-25T07:49:46.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:46.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:46.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:46.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:46.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:46.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:46.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:46.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:46.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:46.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:46.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:46.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:46.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:46.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:47.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:47.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:47.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:47.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:47.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:47.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30573575 2024-04-25T07:49:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30573575 '!=' null 2024-04-25T07:49:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:48.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:48.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:48.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:48.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:48.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:48.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:48.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:48.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:48.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:48.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:48.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:48.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:48.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:49.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:49.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:49.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:49.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:49.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:49.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:49.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:49.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:49.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:50.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24458865 2024-04-25T07:49:50.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24458865 '!=' null 2024-04-25T07:49:50.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:50.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:50.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:50.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:50.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:50.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:50.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:50.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:50.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:50.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:50.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:50.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:50.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:50.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:50.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:51.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:51.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:51.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:51.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:51.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:51.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:51.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:51.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:51.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:51.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24458865 2024-04-25T07:49:51.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24458865 '!=' null 2024-04-25T07:49:51.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:51.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:51.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:51.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:51.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:51.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:51.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:51.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:51.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:51.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:51.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:52.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:52.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:52.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:52.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:53.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:53.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:53.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:53.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:53.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:53.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:53.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:53.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:53.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574272 2024-04-25T07:49:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574272 '!=' null 2024-04-25T07:49:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:53.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:53.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:53.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:53.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:53.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:53.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:53.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:53.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:54.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:54.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:54.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:54.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:54.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:54.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:54.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:54.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:54.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:54.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:54.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:54.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:54.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:55.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574272 2024-04-25T07:49:55.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574272 '!=' null 2024-04-25T07:49:55.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:55.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:55.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:55.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:55.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:55.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:55.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:55.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:55.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:55.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:55.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:56.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:56.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:56.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:56.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:56.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:56.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:56.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574348 2024-04-25T07:49:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574348 '!=' null 2024-04-25T07:49:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:57.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:57.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:57.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:57.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:57.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:49:57.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:49:57.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:49:57.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:49:57.997 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:49:58.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:49:58.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:49:58.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:49:58.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:49:58.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:49:58.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:49:58.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:49:58.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:49:58.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:49:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575137 2024-04-25T07:49:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575137 '!=' null 2024-04-25T07:49:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:49:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:49:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:49:59.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:49:59.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:49:59.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:49:59.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:49:59.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:49:59.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:49:59.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:49:59.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:00.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:00.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:00.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:00.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:00.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:00.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:00.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:00.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:00.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:00.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:00.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:00.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:00.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:01.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460112 2024-04-25T07:50:01.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460112 '!=' null 2024-04-25T07:50:01.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:01.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:01.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:01.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:01.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:01.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:01.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:01.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:01.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:01.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:01.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:02.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:02.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:02.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:02.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:02.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:02.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:02.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:02.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:02.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:02.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:02.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:02.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:02.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:03.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575167 2024-04-25T07:50:03.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575167 '!=' null 2024-04-25T07:50:03.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:03.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:03.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:03.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:03.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:03.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:03.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:03.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:03.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:03.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:03.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:03.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:03.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:03.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:03.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:04.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:04.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:04.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:04.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:04.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575069 2024-04-25T07:50:04.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575069 '!=' null 2024-04-25T07:50:04.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:04.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:04.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:05.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:05.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:05.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:05.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:05.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:05.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:05.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:05.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:05.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:05.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:05.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:05.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:06.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:06.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:06.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:06.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:06.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:06.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:06.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:06.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:06.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:06.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575428 2024-04-25T07:50:06.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575428 '!=' null 2024-04-25T07:50:06.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:06.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:06.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:06.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:06.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:06.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:06.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:06.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:06.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:06.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:06.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:07.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:07.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:07.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:07.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:07.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:07.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:07.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:07.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:07.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:07.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:07.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:07.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:07.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:08.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575306 2024-04-25T07:50:08.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575306 '!=' null 2024-04-25T07:50:08.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:08.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:08.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:08.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:08.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:08.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:08.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:08.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:08.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:08.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:08.549 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:08.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:08.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:08.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:08.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:09.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:09.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:09.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:09.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:09.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:09.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:09.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:09.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:09.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:10.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460372 2024-04-25T07:50:10.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460372 '!=' null 2024-04-25T07:50:10.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:10.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:10.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:10.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:10.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:10.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:10.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:10.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:10.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:10.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:10.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:10.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:10.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:10.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:10.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:11.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:11.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:11.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:11.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:11.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:11.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:11.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:11.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:11.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:11.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575351 2024-04-25T07:50:11.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575351 '!=' null 2024-04-25T07:50:11.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:11.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:11.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:11.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:11.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:11.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:11.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:11.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:11.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:11.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:11.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:12.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:12.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:12.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:12.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:12.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:12.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:12.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:12.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:12.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:12.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:12.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:12.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:13.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575653 2024-04-25T07:50:13.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575653 '!=' null 2024-04-25T07:50:13.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:13.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:13.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:13.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:13.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:13.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:13.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:13.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:13.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:13.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:13.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:14.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:14.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:14.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:14.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:14.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:14.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:14.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:14.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:14.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:14.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:14.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:14.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:14.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:15.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575653 2024-04-25T07:50:15.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575653 '!=' null 2024-04-25T07:50:15.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:15.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:15.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:15.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:15.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:15.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:15.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:15.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:15.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:15.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:15.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:16.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:16.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:16.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:16.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:16.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:16.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:16.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:16.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:16.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:16.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:16.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:16.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:16.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:17.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575503 2024-04-25T07:50:17.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575503 '!=' null 2024-04-25T07:50:17.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:17.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:17.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:17.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:17.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:17.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:17.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:17.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:17.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:17.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:17.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:17.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:17.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:17.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:17.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:18.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:18.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:18.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:18.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:18.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:18.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:18.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:18.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:18.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:19.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574582 2024-04-25T07:50:19.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574582 '!=' null 2024-04-25T07:50:19.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:19.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:19.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:19.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:19.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:19.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:19.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:19.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:19.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:19.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:19.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:19.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:19.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:19.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:19.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:20.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:20.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:20.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:20.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:20.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:20.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:20.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:20.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:20.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:21.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459731 2024-04-25T07:50:21.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459731 '!=' null 2024-04-25T07:50:21.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:21.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:21.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:21.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:21.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:21.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:21.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:21.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:21.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:21.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:21.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:21.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:21.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:21.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:21.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:22.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:22.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:22.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:22.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:22.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574422 2024-04-25T07:50:22.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574422 '!=' null 2024-04-25T07:50:22.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:22.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:22.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:23.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:23.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:23.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:23.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:23.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:23.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:23.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:23.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:23.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:23.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:23.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:23.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:24.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:24.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:24.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:24.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:24.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:24.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575133 2024-04-25T07:50:24.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575133 '!=' null 2024-04-25T07:50:24.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:24.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:24.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:24.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:24.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:24.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:24.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:24.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:24.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:24.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:24.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:25.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:25.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:25.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:25.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:25.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:25.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:25.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:25.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:25.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:25.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:25.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:25.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:26.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:26.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30574879 2024-04-25T07:50:26.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30574879 '!=' null 2024-04-25T07:50:26.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:26.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:26.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:26.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:26.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:26.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:26.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:26.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:26.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:26.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:26.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:27.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:27.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:27.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:27.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:27.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:27.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:27.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:28.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701657 2024-04-25T07:50:28.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701657 '!=' null 2024-04-25T07:50:28.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:28.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:28.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:28.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:28.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:28.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:28.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:28.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:28.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:28.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:28.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:28.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:28.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:29.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:29.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:29.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:29.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:29.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:29.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:30.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586815 2024-04-25T07:50:30.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586815 '!=' null 2024-04-25T07:50:30.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:30.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:30.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:30.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:30.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:30.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:30.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:30.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:30.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:30.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:30.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:31.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:31.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:31.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:31.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:31.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:31.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:31.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:31.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:31.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:31.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:31.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:31.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:31.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:32.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701985 2024-04-25T07:50:32.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701985 '!=' null 2024-04-25T07:50:32.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:32.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:32.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:32.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:32.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:32.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:32.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:32.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:32.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:32.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:32.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:32.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:32.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:32.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:32.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:33.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:33.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:33.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:33.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:33.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:33.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:33.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:33.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:33.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:34.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29701581 2024-04-25T07:50:34.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29701581 '!=' null 2024-04-25T07:50:34.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:34.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:34.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:34.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:34.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:34.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:34.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:34.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:34.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:34.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:34.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:35.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:35.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:35.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:35.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:35.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:35.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:36.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586085 2024-04-25T07:50:36.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586085 '!=' null 2024-04-25T07:50:36.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:36.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:36.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:36.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:36.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:36.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:36.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:36.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:36.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:36.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:36.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:36.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:36.997 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:36.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:36.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:37.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:37.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:37.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:38.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=29700975 2024-04-25T07:50:38.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 29700975 '!=' null 2024-04-25T07:50:38.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:38.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:38.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:38.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:38.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:38.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:38.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:38.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:38.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:38.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:38.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:38.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:38.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:39.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:39.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:39.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:39.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:39.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:39.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:39.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:39.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:39.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:40.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27515580 2024-04-25T07:50:40.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27515580 '!=' null 2024-04-25T07:50:40.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:40.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:40.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:40.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:40.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:40.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:40.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:40.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:40.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:40.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:40.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:40.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:40.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:41.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:41.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:41.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:41.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:41.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:41.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:41.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:41.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:41.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:41.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15722666 2024-04-25T07:50:41.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15722666 '!=' null 2024-04-25T07:50:41.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:41.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:41.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:41.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:41.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:41.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:41.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:41.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:41.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:41.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:41.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:42.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:50:42.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:42.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:42.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:42.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T07:50:42.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:50:42.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:42.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:42.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:42.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:42.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:42.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:42.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:43.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15722666 2024-04-25T07:50:43.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15722666 '!=' null 2024-04-25T07:50:43.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:43.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:43.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:43.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:43.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:43.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:43.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:43.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:43.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:43.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:43.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:43.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:50:43.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:43.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:43.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:44.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T07:50:44.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-04-25T07:50:44.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:50:44.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T07:50:44.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:44.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:44.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:44.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:44.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:44.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:44.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:44.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:44.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:50:44.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:44.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:44.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:45.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T07:50:45.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T07:50:45.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:45.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:45.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:45.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:45.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:45.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:45.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:46.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15384754 2024-04-25T07:50:46.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15384754 '!=' null 2024-04-25T07:50:46.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:46.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:46.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:46.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:46.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:46.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:46.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:46.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:46.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:46.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:46.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:46.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:50:46.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:46.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:46.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T07:50:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T07:50:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:50:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:50:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:50:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:50:47.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:50:47.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:50:47.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:50:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793078 2024-04-25T07:50:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793078 '!=' null 2024-04-25T07:50:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:50:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:50:47.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:50:47.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:50:47.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:50:47.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:50:47.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:50:47.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:50:47.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:50:47.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:50:47.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:50:48.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T07:50:48.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:50:48.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:50:48.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:50:48.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T07:50:48.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:50:48.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:50:48.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:525: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2024-04-25T07:50:48.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: echo TEST PASSED 2024-04-25T07:50:48.629 INFO:tasks.workunit.client.0.smithi088.stdout:TEST PASSED 2024-04-25T07:50:48.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: delete_pool test1 2024-04-25T07:50:48.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T07:50:48.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T07:50:49.166 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test1' does not exist 2024-04-25T07:50:49.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:533: TEST_recovery_pool_priority: delete_pool test2 2024-04-25T07:50:49.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-04-25T07:50:49.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-04-25T07:50:49.803 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test2' does not exist 2024-04-25T07:50:49.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:534: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2024-04-25T07:50:49.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:50:49.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:50:49.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:50:49.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:50:49.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:50:55.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:50:55.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:535: TEST_recovery_pool_priority: return 0 2024-04-25T07:50:55.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-04-25T07:50:55.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T07:50:55.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:50:55.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T07:50:55.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:50:55.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:50:55.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:50:55.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:50:55.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:50:55.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:50:55.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:50:55.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:50:55.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:50:55.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:50:55.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:50:55.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:50:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:50:55.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:50:55.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:55.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:50:55.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:50:55.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T07:50:55.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:50:55.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:55.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:50:55.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202479 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:50:55.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:50:55.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:50:55.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:50:55.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:50:55.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:50:55.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:50:55.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:50:55.201 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:50:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:50:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:50:55.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:50:55.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:55.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:50:55.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:50:55.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:55.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:50:55.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:50:55.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T07:50:55.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:50:55.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:55.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:50:55.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202479 2024-04-25T07:50:55.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:50:55.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:50:55.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-04-25T07:50:55.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:50:55.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:55.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:50:55.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.202479 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_priority td/osd-recovery-prio 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local dir=td/osd-recovery-prio 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:48: TEST_recovery_priority: local pools=10 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: local OSDS=5 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: local max_tries=10 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:52: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-04-25T07:50:55.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:50:55.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:50:55.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:50:55.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-04-25T07:50:55.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-04-25T07:50:55.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:50:55.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:50:55.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:50:55.687 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:50:55.688 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:55.688 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:50:55.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:50:55.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:50:55.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:50:55.908 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:50:55.908 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:50:55.908 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:50:55.908 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:50:55.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:50:55.909 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:50:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:50:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:50:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:50:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:50:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-mon.a.asok 2024-04-25T07:50:55.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:50:55.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202479/ceph-mon.a.asok config get fsid 2024-04-25T07:50:56.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:50:56.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:50:56.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:50:56.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:50:56.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:50:56.028 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:50:56.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:50:56.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:50:56.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:50:56.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:56.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:50:56.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-mon.a.asok 2024-04-25T07:50:56.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:50:56.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202479/ceph-mon.a.asok config get mon_host 2024-04-25T07:50:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2024-04-25T07:50:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-04-25T07:50:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:50:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:50:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:50:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-04-25T07:50:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:50:57.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:50:57.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:50:57.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:50:57.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:50:57.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:57.038 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:50:57.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:50:57.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:50:57.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:50:57.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:54: TEST_recovery_priority: export CEPH_ARGS 2024-04-25T07:50:57.075 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: expr 5 - 1 2024-04-25T07:50:57.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: seq 0 4 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-04-25T07:50:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:50:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-04-25T07:50:57.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:50:57.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba8af0e1-89e2-4d7a-9ab0-91755398f0e1 2024-04-25T07:50:57.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ba8af0e1-89e2-4d7a-9ab0-91755398f0e1' 2024-04-25T07:50:57.083 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 ba8af0e1-89e2-4d7a-9ab0-91755398f0e1 2024-04-25T07:50:57.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:50:57.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhCypmWGNJBhAAGQYQfFKCcwzupmOuavH42w== 2024-04-25T07:50:57.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhCypmWGNJBhAAGQYQfFKCcwzupmOuavH42w=="}' 2024-04-25T07:50:57.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba8af0e1-89e2-4d7a-9ab0-91755398f0e1 -i td/osd-recovery-prio/0/new.json 2024-04-25T07:50:57.393 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:50:57.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-04-25T07:50:57.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhCypmWGNJBhAAGQYQfFKCcwzupmOuavH42w== --osd-uuid ba8af0e1-89e2-4d7a-9ab0-91755398f0e1 2024-04-25T07:50:57.446 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:50:57.448+0000 7f8dce64f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:57.462 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:50:57.464+0000 7f8dce64f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:57.463 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:50:57.464+0000 7f8dce64f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:57.463 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:50:57.464+0000 7f8dce64f7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T07:50:57.464 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:50:57.464+0000 7f8dce64f7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T07:50:57.464 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:50:57.464+0000 7f8dce64f7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T07:50:57.502 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:50:57.504+0000 7f8dce64f7c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-04-25T07:51:03.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-04-25T07:51:03.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:51:03.616 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T07:51:03.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:51:03.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:51:04.162 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T07:51:04.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:51:04.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:51:04.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:04.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:51:04.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:51:04.198 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:04.200+0000 7ffa2edbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:04.234 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:04.236+0000 7ffa2edbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:04.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:04.236+0000 7ffa2edbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:04.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:51:04.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:04.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:51:04.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:04.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:04.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:04.591 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:51:04.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:04.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:04.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:04.796 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:04.796+0000 7ffa2edbb7c0 -1 Falling back to public interface 2024-04-25T07:51:04.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:05.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:05.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:05.999 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:51:06.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:06.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:06.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:06.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:07.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:07.312+0000 7ffa2edbb7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:51:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:07.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:07.404 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:51:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:07.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:08.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:08.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:08.956 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:51:08.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:08.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:08.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:09.390 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/355326589,v1:127.0.0.1:6803/355326589] [v2:127.0.0.1:6804/355326589,v1:127.0.0.1:6805/355326589] exists,up ba8af0e1-89e2-4d7a-9ab0-91755398f0e1 2024-04-25T07:51:09.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:09.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:09.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-04-25T07:51:09.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:51:09.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:51:09.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01426004-4704-4bea-89e7-4bdbfd7d1764 2024-04-25T07:51:09.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 01426004-4704-4bea-89e7-4bdbfd7d1764' 2024-04-25T07:51:09.397 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 01426004-4704-4bea-89e7-4bdbfd7d1764 2024-04-25T07:51:09.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:51:09.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtCypmPSbJGBAAFwWihibSzbZq8D82ogCffg== 2024-04-25T07:51:09.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtCypmPSbJGBAAFwWihibSzbZq8D82ogCffg=="}' 2024-04-25T07:51:09.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01426004-4704-4bea-89e7-4bdbfd7d1764 -i td/osd-recovery-prio/1/new.json 2024-04-25T07:51:09.918 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:51:09.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-04-25T07:51:09.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtCypmPSbJGBAAFwWihibSzbZq8D82ogCffg== --osd-uuid 01426004-4704-4bea-89e7-4bdbfd7d1764 2024-04-25T07:51:09.972 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:09.972+0000 7f1a753ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:09.999 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:10.000+0000 7f1a753ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:10.001 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:10.000+0000 7f1a753ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:10.001 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:10.000+0000 7f1a753ea7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T07:51:10.001 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:10.000+0000 7f1a753ea7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T07:51:10.001 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:10.000+0000 7f1a753ea7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T07:51:10.099 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:10.100+0000 7f1a753ea7c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-04-25T07:51:15.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-04-25T07:51:15.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:51:15.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:51:15.961 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T07:51:15.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:51:16.535 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T07:51:16.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:51:16.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:16.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:51:16.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:51:16.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:51:16.569 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:16.568+0000 7f5fb3efe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:16.579 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:16.580+0000 7f5fb3efe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:16.581 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:16.580+0000 7f5fb3efe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:16.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:51:16.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:16.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:51:16.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:16.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:16.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:16.985 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:51:16.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:16.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:16.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:17.167 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:17.168+0000 7f5fb3efe7c0 -1 Falling back to public interface 2024-04-25T07:51:17.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:18.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:18.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:18.403 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:51:18.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:18.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:18.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:18.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:19.721 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:19.720+0000 7f5fb3efe7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:51:19.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:19.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:19.825 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:51:19.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:19.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:19.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:20.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:21.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:21.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:21.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:21.343 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:51:21.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:21.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:21.793 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1963977235,v1:127.0.0.1:6811/1963977235] [v2:127.0.0.1:6812/1963977235,v1:127.0.0.1:6813/1963977235] exists,up 01426004-4704-4bea-89e7-4bdbfd7d1764 2024-04-25T07:51:21.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:21.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:21.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:51:21.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:51:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2024-04-25T07:51:21.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:51:21.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9659008-b546-485f-964e-3ad7d0d6a178 2024-04-25T07:51:21.800 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 d9659008-b546-485f-964e-3ad7d0d6a178 2024-04-25T07:51:21.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d9659008-b546-485f-964e-3ad7d0d6a178' 2024-04-25T07:51:21.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:51:21.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5CypmxSs0MRAA832lGN6PBLyVTgI0Hh+A2Q== 2024-04-25T07:51:21.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5CypmxSs0MRAA832lGN6PBLyVTgI0Hh+A2Q=="}' 2024-04-25T07:51:21.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9659008-b546-485f-964e-3ad7d0d6a178 -i td/osd-recovery-prio/2/new.json 2024-04-25T07:51:22.389 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:51:22.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2024-04-25T07:51:22.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5CypmxSs0MRAA832lGN6PBLyVTgI0Hh+A2Q== --osd-uuid d9659008-b546-485f-964e-3ad7d0d6a178 2024-04-25T07:51:22.441 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:22.440+0000 7f9d546a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:22.456 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:22.456+0000 7f9d546a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:22.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:22.456+0000 7f9d546a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:22.460+0000 7f9d546a07c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T07:51:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:22.460+0000 7f9d546a07c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T07:51:22.458 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:22.460+0000 7f9d546a07c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T07:51:22.495 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:22.496+0000 7f9d546a07c0 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2024-04-25T07:51:28.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2024-04-25T07:51:28.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:51:28.676 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T07:51:28.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:51:28.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:51:29.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:51:29.239 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T07:51:29.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:29.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:51:29.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:51:29.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:51:29.272 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:29.271+0000 7f38641967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:29.294 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:29.295+0000 7f38641967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:29.295 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:29.295+0000 7f38641967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:29.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:51:29.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:29.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:51:29.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:29.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:29.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:29.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:29.724 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:51:29.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:29.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:29.843 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:29.843+0000 7f38641967c0 -1 Falling back to public interface 2024-04-25T07:51:30.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:31.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:31.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:31.175 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:51:31.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:31.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:31.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:32.543 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:32.543+0000 7f38641967c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:51:32.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:32.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:32.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:32.595 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:51:32.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:32.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:33.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:34.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:34.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:34.112 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:51:34.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:34.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:34.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:34.532 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4148908272,v1:127.0.0.1:6819/4148908272] [v2:127.0.0.1:6820/4148908272,v1:127.0.0.1:6821/4148908272] exists,up d9659008-b546-485f-964e-3ad7d0d6a178 2024-04-25T07:51:34.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:34.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:51:34.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:51:34.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2024-04-25T07:51:34.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:51:34.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=52ae6bb9-7d93-4d9f-956b-757b7b2b0ff4 2024-04-25T07:51:34.539 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 52ae6bb9-7d93-4d9f-956b-757b7b2b0ff4 2024-04-25T07:51:34.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 52ae6bb9-7d93-4d9f-956b-757b7b2b0ff4' 2024-04-25T07:51:34.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:51:34.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGCypmmJy8IRAAUGuHxzqZ9I6u5VA6GWMefQ== 2024-04-25T07:51:34.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGCypmmJy8IRAAUGuHxzqZ9I6u5VA6GWMefQ=="}' 2024-04-25T07:51:34.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 52ae6bb9-7d93-4d9f-956b-757b7b2b0ff4 -i td/osd-recovery-prio/3/new.json 2024-04-25T07:51:35.095 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:51:35.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2024-04-25T07:51:35.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGCypmmJy8IRAAUGuHxzqZ9I6u5VA6GWMefQ== --osd-uuid 52ae6bb9-7d93-4d9f-956b-757b7b2b0ff4 2024-04-25T07:51:35.148 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:35.147+0000 7f39181537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:35.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:35.151+0000 7f39181537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:35.151 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:35.151+0000 7f39181537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:35.152 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:35.151+0000 7f39181537c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T07:51:35.152 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:35.151+0000 7f39181537c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T07:51:35.152 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:35.151+0000 7f39181537c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T07:51:35.194 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:35.195+0000 7f39181537c0 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2024-04-25T07:51:41.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2024-04-25T07:51:41.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:51:41.348 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T07:51:41.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:51:41.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:51:41.911 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T07:51:41.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:51:41.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:41.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:51:41.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:51:41.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:51:41.944 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:41.943+0000 7f1430b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:41.947 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:41.947+0000 7f1430b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:41.948 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:41.947+0000 7f1430b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:42.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:51:42.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:42.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:51:42.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:42.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:42.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:42.347 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:51:42.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:42.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:42.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:51:42.511 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:42.511+0000 7f1430b987c0 -1 Falling back to public interface 2024-04-25T07:51:42.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:43.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:43.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:43.763 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:51:43.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:43.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:43.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:51:44.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:45.068 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:45.067+0000 7f1430b987c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:51:45.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:45.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:45.234 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:51:45.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:45.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:45.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:51:45.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:46.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:46.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:46.746 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:51:46.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:46.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:51:47.170 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2053924177,v1:127.0.0.1:6827/2053924177] [v2:127.0.0.1:6828/2053924177,v1:127.0.0.1:6829/2053924177] exists,up 52ae6bb9-7d93-4d9f-956b-757b7b2b0ff4 2024-04-25T07:51:47.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:47.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:47.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:47.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T07:51:47.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2024-04-25T07:51:47.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T07:51:47.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:51:47.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:47.173 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:47.174 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:47.174 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:51:47.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:51:47.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' 2024-04-25T07:51:47.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:47.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:51:47.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:51:47.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T07:51:47.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f097fc2-7fd3-42d2-9735-de9425fe6ebf 2024-04-25T07:51:47.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 7f097fc2-7fd3-42d2-9735-de9425fe6ebf' 2024-04-25T07:51:47.177 INFO:tasks.workunit.client.0.smithi088.stdout:add osd4 7f097fc2-7fd3-42d2-9735-de9425fe6ebf 2024-04-25T07:51:47.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:51:47.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTCypm47fCCxAAWBGY4R2L4HtTvANVm3v95Q== 2024-04-25T07:51:47.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTCypm47fCCxAAWBGY4R2L4HtTvANVm3v95Q=="}' 2024-04-25T07:51:47.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f097fc2-7fd3-42d2-9735-de9425fe6ebf -i td/osd-recovery-prio/4/new.json 2024-04-25T07:51:47.661 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T07:51:47.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2024-04-25T07:51:47.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTCypm47fCCxAAWBGY4R2L4HtTvANVm3v95Q== --osd-uuid 7f097fc2-7fd3-42d2-9735-de9425fe6ebf 2024-04-25T07:51:47.710 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:47.707+0000 7f3c4776b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:47.744 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:47.743+0000 7f3c4776b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:47.745 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:47.743+0000 7f3c4776b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:47.746 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:47.747+0000 7f3c4776b7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T07:51:47.746 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:47.747+0000 7f3c4776b7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T07:51:47.746 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:47.747+0000 7f3c4776b7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T07:51:47.868 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:47.867+0000 7f3c4776b7c0 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2024-04-25T07:51:54.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2024-04-25T07:51:54.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:51:54.013 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd4 key to auth repository 2024-04-25T07:51:54.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T07:51:54.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:51:54.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T07:51:54.587 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.4 2024-04-25T07:51:54.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=edb3f5e0-c7a0-4dc3-978d-85d7979a647e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.202479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:54.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:51:54.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:51:54.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:51:54.618 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:54.619+0000 7f596abf37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:54.669 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:54.667+0000 7f596abf37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:54.671 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:54.671+0000 7f596abf37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:55.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T07:51:55.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:55.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T07:51:55.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:55.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:55.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:55.073 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T07:51:55.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:55.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:55.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T07:51:55.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:55.235+0000 7f596abf37c0 -1 Falling back to public interface 2024-04-25T07:51:55.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:56.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:56.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:56.516 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T07:51:56.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:56.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:56.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T07:51:56.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:57.773 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T07:51:57.771+0000 7f596abf37c0 -1 osd.4 0 log_to_monitors true 2024-04-25T07:51:57.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:57.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:57.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:57.996 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T07:51:57.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:57.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T07:51:58.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:59.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:59.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:59.626 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T07:51:59.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:59.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:59.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T07:52:00.041 INFO:tasks.workunit.client.0.smithi088.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3177714796,v1:127.0.0.1:6835/3177714796] [v2:127.0.0.1:6836/3177714796,v1:127.0.0.1:6837/3177714796] exists,up 7f097fc2-7fd3-42d2-9735-de9425fe6ebf 2024-04-25T07:52:00.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:52:00.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:52:00.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:52:00.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: seq 1 10 2024-04-25T07:52:00.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:00.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test1 1 1 2024-04-25T07:52:00.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T07:52:00.653 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test1' already exists 2024-04-25T07:52:00.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:01.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test1 size 2 2024-04-25T07:52:02.485 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 2 2024-04-25T07:52:02.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:02.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test2 1 1 2024-04-25T07:52:02.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-04-25T07:52:03.000 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test2' already exists 2024-04-25T07:52:03.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:04.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test2 size 2 2024-04-25T07:52:04.676 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 2 size to 2 2024-04-25T07:52:04.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:04.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test3 1 1 2024-04-25T07:52:04.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-04-25T07:52:05.287 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test3' already exists 2024-04-25T07:52:05.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:06.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-04-25T07:52:07.505 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 3 size to 2 2024-04-25T07:52:07.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:07.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test4 1 1 2024-04-25T07:52:07.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2024-04-25T07:52:08.254 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test4' already exists 2024-04-25T07:52:08.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:09.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test4 size 2 2024-04-25T07:52:10.224 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 4 size to 2 2024-04-25T07:52:10.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:10.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test5 1 1 2024-04-25T07:52:10.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2024-04-25T07:52:10.804 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test5' already exists 2024-04-25T07:52:10.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:11.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test5 size 2 2024-04-25T07:52:12.428 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 5 size to 2 2024-04-25T07:52:12.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:12.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test6 1 1 2024-04-25T07:52:12.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2024-04-25T07:52:13.400 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test6' already exists 2024-04-25T07:52:13.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:14.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test6 size 2 2024-04-25T07:52:15.168 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 6 size to 2 2024-04-25T07:52:15.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:15.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test7 1 1 2024-04-25T07:52:15.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2024-04-25T07:52:15.735 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test7' already exists 2024-04-25T07:52:15.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:16.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-04-25T07:52:17.548 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 7 size to 2 2024-04-25T07:52:17.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:17.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test8 1 1 2024-04-25T07:52:17.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2024-04-25T07:52:18.512 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test8' already exists 2024-04-25T07:52:18.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:19.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test8 size 2 2024-04-25T07:52:20.272 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 8 size to 2 2024-04-25T07:52:20.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:20.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test9 1 1 2024-04-25T07:52:20.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2024-04-25T07:52:21.181 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test9' already exists 2024-04-25T07:52:21.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:22.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test9 size 2 2024-04-25T07:52:23.318 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 9 size to 2 2024-04-25T07:52:23.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:23.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test10 1 1 2024-04-25T07:52:23.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2024-04-25T07:52:24.226 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test10' already exists 2024-04-25T07:52:24.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:52:25.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test10 size 2 2024-04-25T07:52:26.251 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 10 size to 2 2024-04-25T07:52:26.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:66: TEST_recovery_priority: sleep 5 2024-04-25T07:52:31.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:68: TEST_recovery_priority: wait_for_clean 2024-04-25T07:52:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:52:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:52:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:52:31.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:52:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:52:31.278 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:52:31.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:52:31.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:52:31.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:52:31.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:52:31.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:52:31.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:52:31.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:52:31.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:52:31.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:52:31.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:52:31.833 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:52:31.833 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:52:31.833 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T07:52:31.833 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T07:52:31.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:52:31.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:31.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:52:32.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836499 2024-04-25T07:52:32.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836499 2024-04-25T07:52:32.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499' 2024-04-25T07:52:32.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:32.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:52:32.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-04-25T07:52:32.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-04-25T07:52:32.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672977' 2024-04-25T07:52:32.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:32.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:52:32.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-04-25T07:52:32.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-04-25T07:52:32.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454' 2024-04-25T07:52:32.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:32.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:52:32.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345933 2024-04-25T07:52:32.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345933 2024-04-25T07:52:32.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454 3-85899345933' 2024-04-25T07:52:32.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:32.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T07:52:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182409 2024-04-25T07:52:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182409 2024-04-25T07:52:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509454 3-85899345933 4-107374182409' 2024-04-25T07:52:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:32.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836499 2024-04-25T07:52:32.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:32.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:52:32.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836499 2024-04-25T07:52:32.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:32.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836499 2024-04-25T07:52:32.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-04-25T07:52:32.638 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836499 2024-04-25T07:52:32.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:52:33.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836497 -lt 21474836499 2024-04-25T07:52:33.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:52:34.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:52:34.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:52:34.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836499 2024-04-25T07:52:34.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:34.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-04-25T07:52:34.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:34.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:52:34.463 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-04-25T07:52:34.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:34.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-04-25T07:52:34.466 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672977 2024-04-25T07:52:34.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-25T07:52:34.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:52:34.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672977 2024-04-25T07:52:34.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:34.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:34.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-04-25T07:52:34.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:52:34.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-04-25T07:52:34.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:34.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-04-25T07:52:34.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-25T07:52:34.866 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509454 2024-04-25T07:52:34.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:52:35.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-25T07:52:35.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:35.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345933 2024-04-25T07:52:35.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:35.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:52:35.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345933 2024-04-25T07:52:35.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:35.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345933 2024-04-25T07:52:35.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-04-25T07:52:35.298 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345933 2024-04-25T07:52:35.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:52:35.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345933 -lt 85899345933 2024-04-25T07:52:35.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:35.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182409 2024-04-25T07:52:35.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:35.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T07:52:35.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182409 2024-04-25T07:52:35.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:35.717 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182409 2024-04-25T07:52:35.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182409 2024-04-25T07:52:35.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-04-25T07:52:35.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T07:52:36.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182410 -lt 107374182409 2024-04-25T07:52:36.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:52:36.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:52:36.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:52:36.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-04-25T07:52:36.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:52:36.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:52:36.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:52:36.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:52:36.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:52:36.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:52:36.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:52:37.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-04-25T07:52:37.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:52:37.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:52:37.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:52:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-04-25T07:52:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:52:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:52:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:70: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:26.927774+0000 0'0 80:23 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:52:23.770724+0000 0'0 2024-04-25T07:52:23.770724+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:38:02.041093+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:23.865092+0000 0'0 80:35 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:52:20.896284+0000 0'0 2024-04-25T07:52:20.896284+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:11:38.422059+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:21.255660+0000 0'0 80:43 [3,4] 3 [3,4] 3 0'0 2024-04-25T07:52:18.105244+0000 0'0 2024-04-25T07:52:18.105244+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:59:47.871369+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:18.249074+0000 0'0 80:55 [1,4] 1 [1,4] 1 0'0 2024-04-25T07:52:15.586094+0000 0'0 2024-04-25T07:52:15.586094+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:29:21.642466+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:16.617276+0000 0'0 80:55 [4,2] 4 [4,2] 4 0'0 2024-04-25T07:52:13.068845+0000 0'0 2024-04-25T07:52:13.068845+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:00:08.039009+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:13.205947+0000 0'0 80:69 [3,4] 3 [3,4] 3 0'0 2024-04-25T07:52:10.601488+0000 0'0 2024-04-25T07:52:10.601488+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:01:01.382112+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:10.455265+0000 0'0 80:76 [4,0] 4 [4,0] 4 0'0 2024-04-25T07:52:08.050589+0000 0'0 2024-04-25T07:52:08.050589+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:56:11.230813+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:08.411224+0000 0'0 80:88 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:52:05.141446+0000 0'0 2024-04-25T07:52:05.141446+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:43:15.075098+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:07.176523+0000 0'0 80:92 [3,1] 3 [3,1] 3 0'0 2024-04-25T07:52:02.857646+0000 0'0 2024-04-25T07:52:02.857646+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:23:22.878447+0000 0 0 2024-04-25T07:52:37.973 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:52:03.380981+0000 0'0 80:105 [1,0] 1 [1,0] 1 0'0 2024-04-25T07:52:00.405703+0000 0'0 2024-04-25T07:52:00.405703+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:29:52.928491+0000 0 0 2024-04-25T07:52:37.974 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:52:37.974 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:52:37.974 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:52:37.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local PG1 2024-04-25T07:52:37.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local POOLNUM1 2024-04-25T07:52:37.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:76: TEST_recovery_priority: local pool1 2024-04-25T07:52:37.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local chk_osd1_1 2024-04-25T07:52:37.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local chk_osd1_2 2024-04-25T07:52:37.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local PG2 2024-04-25T07:52:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:81: TEST_recovery_priority: local POOLNUM2 2024-04-25T07:52:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local pool2 2024-04-25T07:52:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local chk_osd2 2024-04-25T07:52:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:85: TEST_recovery_priority: local PG3 2024-04-25T07:52:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: local POOLNUM3 2024-04-25T07:52:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:87: TEST_recovery_priority: local pool3 2024-04-25T07:52:37.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: seq 1 10 2024-04-25T07:52:37.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:37.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 1.0 --format=json 2024-04-25T07:52:37.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T07:52:38.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:52:38.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T07:52:38.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z '' ']' 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: PG1=1.0 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: POOLNUM1=1 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: pool1=test1 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:99: TEST_recovery_priority: chk_osd1_1=1 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: chk_osd1_2=0 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:38.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 2.0 --format=json 2024-04-25T07:52:38.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T07:52:38.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:52:38.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-04-25T07:52:38.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:52:38.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=1 2024-04-25T07:52:38.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T07:52:38.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2024-04-25T07:52:38.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2024-04-25T07:52:38.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:38.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 3.0 --format=json 2024-04-25T07:52:38.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T07:52:39.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:52:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T07:52:39.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:52:39.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-04-25T07:52:39.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T07:52:39.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2024-04-25T07:52:39.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: PG2=3.0 2024-04-25T07:52:39.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: POOLNUM2=3 2024-04-25T07:52:39.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:105: TEST_recovery_priority: pool2=test3 2024-04-25T07:52:39.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: chk_osd2=2 2024-04-25T07:52:39.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:39.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 4.0 --format=json 2024-04-25T07:52:39.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T07:52:39.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:52:39.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-04-25T07:52:39.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:52:39.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-04-25T07:52:39.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T07:52:39.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2024-04-25T07:52:39.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2024-04-25T07:52:39.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:39.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 5.0 --format=json 2024-04-25T07:52:39.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T07:52:40.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:52:40.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-04-25T07:52:40.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:52:40.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-04-25T07:52:40.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T07:52:40.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2024-04-25T07:52:40.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-04-25T07:52:40.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:40.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 6.0 --format=json 2024-04-25T07:52:40.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T07:52:40.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:52:40.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-04-25T07:52:40.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:52:40.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-04-25T07:52:40.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T07:52:40.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2024-04-25T07:52:40.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2024-04-25T07:52:40.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:40.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 7.0 --format=json 2024-04-25T07:52:40.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T07:52:40.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T07:52:40.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T07:52:40.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T07:52:40.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-04-25T07:52:40.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T07:52:40.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2024-04-25T07:52:40.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-04-25T07:52:40.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: PG3=7.0 2024-04-25T07:52:40.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:110: TEST_recovery_priority: POOLNUM3=7 2024-04-25T07:52:40.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:111: TEST_recovery_priority: pool3=test7 2024-04-25T07:52:40.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: break 2024-04-25T07:52:40.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:115: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2024-04-25T07:52:40.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:117: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2024-04-25T07:52:40.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: seq 1 10 2024-04-25T07:52:40.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:40.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2024-04-25T07:52:40.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:40.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2024-04-25T07:52:40.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test2 2024-04-25T07:52:40.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-04-25T07:52:40.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-04-25T07:52:41.503 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test2' does not exist 2024-04-25T07:52:41.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:41.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2024-04-25T07:52:41.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:41.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2024-04-25T07:52:41.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test4 2024-04-25T07:52:41.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2024-04-25T07:52:41.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2024-04-25T07:52:42.133 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test4' does not exist 2024-04-25T07:52:42.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:42.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2024-04-25T07:52:42.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test5 2024-04-25T07:52:42.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2024-04-25T07:52:42.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2024-04-25T07:52:42.769 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test5' does not exist 2024-04-25T07:52:42.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:42.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2024-04-25T07:52:42.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test6 2024-04-25T07:52:42.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2024-04-25T07:52:42.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2024-04-25T07:52:43.397 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test6' does not exist 2024-04-25T07:52:43.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:43.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2024-04-25T07:52:43.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:43.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2024-04-25T07:52:43.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test8 2024-04-25T07:52:43.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2024-04-25T07:52:43.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2024-04-25T07:52:44.113 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test8' does not exist 2024-04-25T07:52:44.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:44.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2024-04-25T07:52:44.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test9 2024-04-25T07:52:44.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2024-04-25T07:52:44.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2024-04-25T07:52:44.656 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test9' does not exist 2024-04-25T07:52:44.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T07:52:44.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2024-04-25T07:52:44.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test10 2024-04-25T07:52:44.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2024-04-25T07:52:44.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2024-04-25T07:52:45.261 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test10' does not exist 2024-04-25T07:52:45.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:131: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2024-04-25T07:52:46.144 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 3 size to 1 2024-04-25T07:52:46.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2024-04-25T07:52:47.141 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 7 size to 1 2024-04-25T07:52:47.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: wait_for_clean 2024-04-25T07:52:47.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:52:47.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:52:47.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:52:47.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:52:47.164 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:52:47.164 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:52:47.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:52:47.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:52:47.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:52:47.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:52:47.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:52:47.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:52:47.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:52:47.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:52:47.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:52:47.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:52:47.814 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:52:47.814 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:52:47.814 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T07:52:47.814 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T07:52:47.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:52:47.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:47.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:52:47.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-04-25T07:52:47.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-04-25T07:52:47.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-04-25T07:52:47.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:47.974 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:52:48.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-04-25T07:52:48.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-04-25T07:52:48.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672982' 2024-04-25T07:52:48.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:48.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:52:48.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509459 2024-04-25T07:52:48.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509459 2024-04-25T07:52:48.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-64424509459' 2024-04-25T07:52:48.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:48.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:52:48.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345938 2024-04-25T07:52:48.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345938 2024-04-25T07:52:48.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-64424509459 3-85899345938' 2024-04-25T07:52:48.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:48.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T07:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182414 2024-04-25T07:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182414 2024-04-25T07:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-64424509459 3-85899345938 4-107374182414' 2024-04-25T07:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-04-25T07:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:52:48.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-04-25T07:52:48.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:48.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-04-25T07:52:48.600 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836504 2024-04-25T07:52:48.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-25T07:52:48.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:52:49.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836504 2024-04-25T07:52:49.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:52:50.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:52:50.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:52:50.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836504 2024-04-25T07:52:50.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:50.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-04-25T07:52:50.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:50.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:52:50.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-04-25T07:52:50.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:50.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-04-25T07:52:50.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-25T07:52:50.477 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672982 2024-04-25T07:52:50.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:52:50.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-25T07:52:50.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:50.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509459 2024-04-25T07:52:50.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:50.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:52:50.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509459 2024-04-25T07:52:50.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:50.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509459 2024-04-25T07:52:50.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-04-25T07:52:50.890 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509459 2024-04-25T07:52:50.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:52:51.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509460 -lt 64424509459 2024-04-25T07:52:51.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:51.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:51.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345938 2024-04-25T07:52:51.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:52:51.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345938 2024-04-25T07:52:51.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345938 2024-04-25T07:52:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2024-04-25T07:52:51.297 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345938 2024-04-25T07:52:51.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:52:51.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345939 -lt 85899345938 2024-04-25T07:52:51.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:51.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182414 2024-04-25T07:52:51.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:51.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T07:52:51.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182414 2024-04-25T07:52:51.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:51.723 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182414 2024-04-25T07:52:51.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182414 2024-04-25T07:52:51.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-04-25T07:52:51.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T07:52:52.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182415 -lt 107374182414 2024-04-25T07:52:52.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:52:52.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:52:52.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:52:52.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T07:52:52.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:52:52.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:52:52.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:52:52.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:52:52.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:52:52.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:52:52.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:52:53.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T07:52:53.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:52:53.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:52:53.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:52:53.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T07:52:53.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:52:53.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:52:53.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:135: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-04-25T07:52:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:10+0 records in 2024-04-25T07:52:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:10+0 records out 2024-04-25T07:52:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0383289 s, 274 MB/s 2024-04-25T07:52:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: p=1 2024-04-25T07:52:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T07:52:53.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T07:52:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-04-25T07:52:54.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:54.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-04-25T07:52:55.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:55.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-04-25T07:52:55.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:55.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-04-25T07:52:56.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:56.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-04-25T07:52:57.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:57.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-04-25T07:52:57.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:57.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-04-25T07:52:58.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:58.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-04-25T07:52:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-04-25T07:52:59.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:52:59.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-04-25T07:53:00.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:00.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-04-25T07:53:01.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:01.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-04-25T07:53:01.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:01.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-04-25T07:53:02.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:02.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-04-25T07:53:03.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:03.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-04-25T07:53:03.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:03.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-04-25T07:53:04.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:04.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-04-25T07:53:05.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:05.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-04-25T07:53:05.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:05.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-04-25T07:53:06.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:06.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-04-25T07:53:07.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:07.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-04-25T07:53:07.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:07.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-04-25T07:53:08.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:08.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-04-25T07:53:09.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:09.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-04-25T07:53:09.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:09.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-04-25T07:53:10.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:10.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-04-25T07:53:11.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:11.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-04-25T07:53:11.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:11.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-04-25T07:53:12.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:12.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-04-25T07:53:13.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:13.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-04-25T07:53:13.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:13.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-04-25T07:53:14.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:14.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-04-25T07:53:15.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:15.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-04-25T07:53:15.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:15.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-04-25T07:53:16.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:16.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-04-25T07:53:17.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:17.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-04-25T07:53:17.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:17.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-04-25T07:53:18.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:18.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-04-25T07:53:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-04-25T07:53:19.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:19.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-04-25T07:53:20.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:20.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-04-25T07:53:21.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:21.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-04-25T07:53:21.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:21.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-04-25T07:53:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-04-25T07:53:23.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:23.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-04-25T07:53:23.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:23.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-04-25T07:53:24.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:24.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-04-25T07:53:24.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:24.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-04-25T07:53:25.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:25.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-04-25T07:53:26.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:26.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-04-25T07:53:27.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:27.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-04-25T07:53:27.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:27.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-04-25T07:53:28.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:28.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-04-25T07:53:28.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:28.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-04-25T07:53:29.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:29.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-04-25T07:53:30.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:30.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-04-25T07:53:31.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:31.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-04-25T07:53:31.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:31.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-04-25T07:53:32.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:32.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-04-25T07:53:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-04-25T07:53:33.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:33.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-04-25T07:53:34.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:34.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-04-25T07:53:35.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:35.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-04-25T07:53:35.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:35.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-04-25T07:53:36.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:36.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-04-25T07:53:37.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:37.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-04-25T07:53:37.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:37.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-04-25T07:53:38.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:38.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-04-25T07:53:39.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:39.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-04-25T07:53:39.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:39.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-04-25T07:53:40.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:40.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-04-25T07:53:41.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:41.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-04-25T07:53:41.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:41.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-04-25T07:53:42.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:42.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-04-25T07:53:43.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:43.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-04-25T07:53:43.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:43.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-04-25T07:53:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-04-25T07:53:45.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:45.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-04-25T07:53:45.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:45.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-04-25T07:53:46.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:46.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-04-25T07:53:47.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:47.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-04-25T07:53:47.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:47.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-04-25T07:53:48.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:48.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-04-25T07:53:49.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:49.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-04-25T07:53:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-04-25T07:53:50.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:50.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-04-25T07:53:51.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:51.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-04-25T07:53:51.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:51.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-04-25T07:53:52.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:52.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-04-25T07:53:53.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:53.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-04-25T07:53:53.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:53.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-04-25T07:53:54.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:54.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-04-25T07:53:55.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:55.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-04-25T07:53:55.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:55.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-04-25T07:53:56.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:56.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-04-25T07:53:57.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:57.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-04-25T07:53:57.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:57.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-04-25T07:53:58.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:58.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-04-25T07:53:59.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:59.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-04-25T07:53:59.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:53:59.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-04-25T07:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:00.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-04-25T07:54:01.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:01.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-04-25T07:54:02.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:02.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-04-25T07:54:02.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:02.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-04-25T07:54:03.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:03.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-04-25T07:54:04.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:04.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-04-25T07:54:04.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:04.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-04-25T07:54:05.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:05.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-04-25T07:54:06.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:06.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-04-25T07:54:06.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:06.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-04-25T07:54:07.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:07.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-04-25T07:54:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-04-25T07:54:08.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:08.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-04-25T07:54:09.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:09.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-04-25T07:54:10.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:10.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-04-25T07:54:11.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:11.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-04-25T07:54:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:11.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-04-25T07:54:12.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:12.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-04-25T07:54:12.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:12.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-04-25T07:54:13.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:13.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-04-25T07:54:14.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:14.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-04-25T07:54:15.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:15.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-04-25T07:54:15.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:15.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-04-25T07:54:16.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:16.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-04-25T07:54:17.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:17.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-04-25T07:54:17.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:17.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-04-25T07:54:18.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:18.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-04-25T07:54:19.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:19.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-04-25T07:54:19.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:19.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-04-25T07:54:20.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:20.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-04-25T07:54:21.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:21.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-04-25T07:54:21.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:21.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-04-25T07:54:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:22.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-04-25T07:54:23.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:23.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-04-25T07:54:23.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:23.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-04-25T07:54:24.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:24.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-04-25T07:54:25.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:25.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-04-25T07:54:25.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:25.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-04-25T07:54:26.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:26.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-04-25T07:54:27.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:27.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-04-25T07:54:27.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:27.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-04-25T07:54:28.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:28.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-04-25T07:54:29.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:29.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-04-25T07:54:29.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:29.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-04-25T07:54:30.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:30.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-04-25T07:54:31.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:31.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-04-25T07:54:31.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:31.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-04-25T07:54:32.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:32.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-04-25T07:54:33.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:33.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-04-25T07:54:34.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:34.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-04-25T07:54:34.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:34.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-04-25T07:54:35.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:35.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-04-25T07:54:35.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:35.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-04-25T07:54:36.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:36.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-04-25T07:54:37.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:37.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-04-25T07:54:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:38.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-04-25T07:54:38.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:38.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-04-25T07:54:39.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:39.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-04-25T07:54:40.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:40.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-04-25T07:54:40.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:40.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-04-25T07:54:41.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:41.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-04-25T07:54:42.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:42.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-04-25T07:54:42.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-04-25T07:54:43.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:43.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-04-25T07:54:44.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:44.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-04-25T07:54:44.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:44.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-04-25T07:54:45.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:45.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-04-25T07:54:46.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:46.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-04-25T07:54:47.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:47.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-04-25T07:54:47.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:47.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-04-25T07:54:48.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:48.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-04-25T07:54:49.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:49.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-04-25T07:54:49.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:49.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-04-25T07:54:50.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:50.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-04-25T07:54:51.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:51.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-04-25T07:54:51.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:51.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-04-25T07:54:52.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:52.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-04-25T07:54:53.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:53.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-04-25T07:54:53.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:53.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-04-25T07:54:54.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:54.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-04-25T07:54:55.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:55.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-04-25T07:54:56.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:56.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-04-25T07:54:56.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:56.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-04-25T07:54:57.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:57.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-04-25T07:54:58.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:58.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-04-25T07:54:58.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:58.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-04-25T07:54:59.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:54:59.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-04-25T07:55:00.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:00.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-04-25T07:55:00.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:00.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-04-25T07:55:01.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:01.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-04-25T07:55:02.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:02.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-04-25T07:55:02.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:02.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-04-25T07:55:03.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:03.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-04-25T07:55:04.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:04.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-04-25T07:55:04.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:04.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-04-25T07:55:05.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:05.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-04-25T07:55:06.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:06.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-04-25T07:55:07.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:07.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-04-25T07:55:07.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:07.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-04-25T07:55:08.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:08.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-04-25T07:55:09.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 1 + 1 2024-04-25T07:55:09.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=2 2024-04-25T07:55:09.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T07:55:09.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T07:55:09.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:09.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2024-04-25T07:55:09.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:09.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2024-04-25T07:55:10.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:10.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2024-04-25T07:55:10.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:10.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2024-04-25T07:55:10.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:10.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2024-04-25T07:55:11.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:11.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2024-04-25T07:55:11.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:11.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2024-04-25T07:55:11.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:11.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2024-04-25T07:55:12.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:12.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2024-04-25T07:55:12.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:12.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2024-04-25T07:55:13.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:13.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2024-04-25T07:55:13.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:13.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2024-04-25T07:55:13.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:13.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2024-04-25T07:55:14.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:14.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2024-04-25T07:55:14.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:14.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2024-04-25T07:55:14.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:14.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2024-04-25T07:55:15.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:15.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2024-04-25T07:55:15.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:15.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2024-04-25T07:55:16.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:16.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2024-04-25T07:55:16.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:16.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2024-04-25T07:55:16.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:16.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2024-04-25T07:55:17.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:17.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2024-04-25T07:55:17.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:17.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2024-04-25T07:55:17.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:17.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2024-04-25T07:55:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:18.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2024-04-25T07:55:18.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:18.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2024-04-25T07:55:19.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:19.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2024-04-25T07:55:19.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:19.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2024-04-25T07:55:19.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:19.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2024-04-25T07:55:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:20.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2024-04-25T07:55:20.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:20.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2024-04-25T07:55:21.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:21.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2024-04-25T07:55:21.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:21.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2024-04-25T07:55:21.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:21.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2024-04-25T07:55:22.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:22.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2024-04-25T07:55:22.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:22.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2024-04-25T07:55:22.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:22.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2024-04-25T07:55:23.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:23.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2024-04-25T07:55:23.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:23.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2024-04-25T07:55:24.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:24.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2024-04-25T07:55:24.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:24.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2024-04-25T07:55:24.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:24.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2024-04-25T07:55:25.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:25.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2024-04-25T07:55:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2024-04-25T07:55:25.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:25.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2024-04-25T07:55:26.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:26.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2024-04-25T07:55:26.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:26.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2024-04-25T07:55:26.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:26.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2024-04-25T07:55:27.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:27.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2024-04-25T07:55:27.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:27.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2024-04-25T07:55:28.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:28.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2024-04-25T07:55:28.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:28.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2024-04-25T07:55:28.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:28.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2024-04-25T07:55:29.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:29.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2024-04-25T07:55:29.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:29.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2024-04-25T07:55:30.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:30.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2024-04-25T07:55:30.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:30.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2024-04-25T07:55:30.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:30.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2024-04-25T07:55:31.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:31.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2024-04-25T07:55:31.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:31.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2024-04-25T07:55:31.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:31.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2024-04-25T07:55:32.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:32.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2024-04-25T07:55:32.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:32.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2024-04-25T07:55:33.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:33.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2024-04-25T07:55:33.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:33.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2024-04-25T07:55:34.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:34.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2024-04-25T07:55:34.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:34.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2024-04-25T07:55:34.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:34.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2024-04-25T07:55:35.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:35.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2024-04-25T07:55:35.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:35.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2024-04-25T07:55:35.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:35.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2024-04-25T07:55:36.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:36.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2024-04-25T07:55:36.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:36.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2024-04-25T07:55:37.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:37.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2024-04-25T07:55:37.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:37.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2024-04-25T07:55:37.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:37.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2024-04-25T07:55:38.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:38.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2024-04-25T07:55:38.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:38.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2024-04-25T07:55:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:39.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2024-04-25T07:55:39.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:39.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2024-04-25T07:55:39.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:39.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2024-04-25T07:55:40.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:40.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2024-04-25T07:55:40.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:40.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2024-04-25T07:55:41.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:41.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2024-04-25T07:55:41.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:41.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2024-04-25T07:55:41.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:41.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2024-04-25T07:55:42.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:42.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2024-04-25T07:55:42.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:42.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2024-04-25T07:55:42.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:42.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2024-04-25T07:55:43.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:43.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2024-04-25T07:55:43.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:43.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2024-04-25T07:55:44.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:44.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2024-04-25T07:55:44.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:44.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2024-04-25T07:55:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2024-04-25T07:55:45.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:45.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2024-04-25T07:55:45.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:45.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2024-04-25T07:55:45.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:45.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2024-04-25T07:55:46.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:46.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2024-04-25T07:55:46.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:46.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2024-04-25T07:55:47.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:47.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2024-04-25T07:55:47.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:47.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2024-04-25T07:55:48.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:48.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2024-04-25T07:55:48.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:48.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2024-04-25T07:55:48.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:48.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2024-04-25T07:55:49.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:49.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2024-04-25T07:55:49.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:49.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2024-04-25T07:55:49.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:49.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2024-04-25T07:55:50.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:50.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2024-04-25T07:55:50.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:50.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2024-04-25T07:55:51.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:51.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2024-04-25T07:55:51.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:51.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2024-04-25T07:55:51.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:51.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2024-04-25T07:55:52.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:52.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2024-04-25T07:55:52.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:52.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2024-04-25T07:55:52.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:52.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2024-04-25T07:55:53.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:53.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2024-04-25T07:55:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2024-04-25T07:55:54.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:54.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2024-04-25T07:55:54.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:54.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2024-04-25T07:55:54.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:54.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2024-04-25T07:55:55.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:55.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2024-04-25T07:55:55.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:55.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2024-04-25T07:55:55.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:55.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2024-04-25T07:55:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:56.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2024-04-25T07:55:56.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:56.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2024-04-25T07:55:57.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2024-04-25T07:55:57.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:57.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2024-04-25T07:55:57.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:57.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2024-04-25T07:55:58.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:58.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2024-04-25T07:55:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2024-04-25T07:55:58.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:58.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2024-04-25T07:55:59.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:59.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2024-04-25T07:55:59.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:55:59.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2024-04-25T07:56:00.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:00.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2024-04-25T07:56:00.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:00.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2024-04-25T07:56:00.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:00.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2024-04-25T07:56:01.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:01.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2024-04-25T07:56:01.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:01.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2024-04-25T07:56:02.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:02.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2024-04-25T07:56:02.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:02.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2024-04-25T07:56:02.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:02.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2024-04-25T07:56:03.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:03.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2024-04-25T07:56:03.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:03.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2024-04-25T07:56:04.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:04.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2024-04-25T07:56:04.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:04.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2024-04-25T07:56:04.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:04.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2024-04-25T07:56:05.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:05.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2024-04-25T07:56:05.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:05.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2024-04-25T07:56:06.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:06.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2024-04-25T07:56:06.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:06.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2024-04-25T07:56:06.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:06.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2024-04-25T07:56:07.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:07.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2024-04-25T07:56:07.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:07.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2024-04-25T07:56:08.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:08.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2024-04-25T07:56:08.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:08.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2024-04-25T07:56:08.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:08.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2024-04-25T07:56:09.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:09.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2024-04-25T07:56:09.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:09.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2024-04-25T07:56:10.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:10.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2024-04-25T07:56:10.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:10.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2024-04-25T07:56:10.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:10.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2024-04-25T07:56:11.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:11.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2024-04-25T07:56:11.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:11.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2024-04-25T07:56:11.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:11.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2024-04-25T07:56:12.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:12.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2024-04-25T07:56:12.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:12.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2024-04-25T07:56:13.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:13.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2024-04-25T07:56:13.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:13.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2024-04-25T07:56:13.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:13.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2024-04-25T07:56:14.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:14.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2024-04-25T07:56:14.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:14.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2024-04-25T07:56:15.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:15.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2024-04-25T07:56:15.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:15.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2024-04-25T07:56:15.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:15.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2024-04-25T07:56:16.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:16.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2024-04-25T07:56:16.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:16.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2024-04-25T07:56:16.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:16.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2024-04-25T07:56:17.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:17.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2024-04-25T07:56:17.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:17.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2024-04-25T07:56:17.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:17.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2024-04-25T07:56:18.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:18.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2024-04-25T07:56:18.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:18.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2024-04-25T07:56:19.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:19.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2024-04-25T07:56:19.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:19.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2024-04-25T07:56:19.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:19.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2024-04-25T07:56:20.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:20.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2024-04-25T07:56:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:20.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2024-04-25T07:56:21.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:21.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2024-04-25T07:56:21.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:21.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2024-04-25T07:56:21.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:21.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2024-04-25T07:56:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2024-04-25T07:56:22.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:22.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2024-04-25T07:56:23.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:23.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2024-04-25T07:56:23.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:23.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2024-04-25T07:56:23.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:23.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2024-04-25T07:56:24.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:24.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2024-04-25T07:56:24.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:24.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2024-04-25T07:56:25.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:25.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2024-04-25T07:56:25.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:25.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2024-04-25T07:56:25.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:25.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2024-04-25T07:56:26.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 2 + 1 2024-04-25T07:56:26.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=3 2024-04-25T07:56:26.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T07:56:26.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T07:56:26.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:26.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2024-04-25T07:56:26.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:26.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2024-04-25T07:56:26.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:26.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2024-04-25T07:56:27.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:27.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2024-04-25T07:56:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2024-04-25T07:56:28.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:28.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2024-04-25T07:56:28.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:28.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2024-04-25T07:56:28.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:28.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2024-04-25T07:56:29.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:29.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2024-04-25T07:56:29.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:29.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2024-04-25T07:56:29.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:29.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2024-04-25T07:56:30.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:30.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2024-04-25T07:56:30.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:30.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2024-04-25T07:56:31.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:31.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2024-04-25T07:56:31.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:31.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2024-04-25T07:56:31.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:31.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2024-04-25T07:56:32.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:32.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2024-04-25T07:56:32.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:32.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2024-04-25T07:56:33.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:33.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2024-04-25T07:56:33.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:33.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2024-04-25T07:56:33.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:33.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2024-04-25T07:56:34.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:34.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2024-04-25T07:56:34.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:34.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2024-04-25T07:56:35.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:35.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2024-04-25T07:56:35.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:35.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2024-04-25T07:56:35.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:35.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2024-04-25T07:56:36.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:36.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2024-04-25T07:56:36.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:36.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2024-04-25T07:56:37.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:37.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2024-04-25T07:56:37.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:37.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2024-04-25T07:56:37.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:37.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2024-04-25T07:56:38.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:38.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2024-04-25T07:56:38.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:38.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2024-04-25T07:56:38.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:38.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2024-04-25T07:56:39.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:39.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2024-04-25T07:56:39.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:39.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2024-04-25T07:56:40.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:40.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2024-04-25T07:56:40.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:40.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2024-04-25T07:56:40.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:40.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2024-04-25T07:56:41.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:41.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2024-04-25T07:56:41.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:41.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2024-04-25T07:56:41.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:41.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2024-04-25T07:56:42.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:42.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2024-04-25T07:56:42.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:42.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2024-04-25T07:56:43.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:43.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2024-04-25T07:56:43.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:43.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2024-04-25T07:56:43.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:43.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2024-04-25T07:56:44.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:44.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2024-04-25T07:56:44.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:44.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2024-04-25T07:56:45.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:45.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2024-04-25T07:56:45.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:45.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2024-04-25T07:56:45.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:45.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2024-04-25T07:56:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2024-04-25T07:56:46.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:46.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2024-04-25T07:56:46.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:46.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2024-04-25T07:56:47.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:47.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2024-04-25T07:56:47.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:47.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2024-04-25T07:56:48.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:48.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2024-04-25T07:56:48.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:48.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2024-04-25T07:56:48.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:48.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2024-04-25T07:56:49.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:49.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2024-04-25T07:56:49.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:49.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2024-04-25T07:56:50.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:50.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2024-04-25T07:56:50.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:50.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2024-04-25T07:56:50.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:50.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2024-04-25T07:56:51.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:51.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2024-04-25T07:56:51.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:51.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2024-04-25T07:56:52.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:52.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2024-04-25T07:56:52.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:52.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2024-04-25T07:56:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2024-04-25T07:56:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2024-04-25T07:56:53.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:53.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2024-04-25T07:56:53.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:53.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2024-04-25T07:56:54.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:54.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2024-04-25T07:56:54.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:54.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2024-04-25T07:56:55.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:55.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2024-04-25T07:56:55.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:55.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2024-04-25T07:56:55.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:55.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2024-04-25T07:56:56.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:56.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2024-04-25T07:56:56.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:56.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2024-04-25T07:56:56.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:56.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2024-04-25T07:56:57.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:57.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2024-04-25T07:56:57.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:57.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2024-04-25T07:56:58.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:58.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2024-04-25T07:56:58.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:58.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2024-04-25T07:56:58.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:58.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2024-04-25T07:56:59.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:59.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2024-04-25T07:56:59.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:59.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2024-04-25T07:56:59.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:56:59.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2024-04-25T07:57:00.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:00.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2024-04-25T07:57:00.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:00.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2024-04-25T07:57:01.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:01.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2024-04-25T07:57:01.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:01.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2024-04-25T07:57:01.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:01.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2024-04-25T07:57:02.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:02.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2024-04-25T07:57:02.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:02.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2024-04-25T07:57:02.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:02.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2024-04-25T07:57:03.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:03.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2024-04-25T07:57:03.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:03.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2024-04-25T07:57:04.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:04.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2024-04-25T07:57:04.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:04.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2024-04-25T07:57:04.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:04.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2024-04-25T07:57:05.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:05.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2024-04-25T07:57:05.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:05.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2024-04-25T07:57:06.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:06.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2024-04-25T07:57:06.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:06.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2024-04-25T07:57:06.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:06.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2024-04-25T07:57:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2024-04-25T07:57:07.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:07.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2024-04-25T07:57:08.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:08.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2024-04-25T07:57:08.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:08.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2024-04-25T07:57:08.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:08.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2024-04-25T07:57:09.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:09.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2024-04-25T07:57:09.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:09.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2024-04-25T07:57:09.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:09.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2024-04-25T07:57:10.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:10.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2024-04-25T07:57:10.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:10.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2024-04-25T07:57:11.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:11.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2024-04-25T07:57:11.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:11.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2024-04-25T07:57:11.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:11.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2024-04-25T07:57:12.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:12.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2024-04-25T07:57:12.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:12.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2024-04-25T07:57:13.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:13.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2024-04-25T07:57:13.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:13.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2024-04-25T07:57:13.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:13.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2024-04-25T07:57:14.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:14.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2024-04-25T07:57:14.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:14.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2024-04-25T07:57:15.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:15.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2024-04-25T07:57:15.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:15.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2024-04-25T07:57:15.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:15.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2024-04-25T07:57:16.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:16.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2024-04-25T07:57:16.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:16.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2024-04-25T07:57:16.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:16.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2024-04-25T07:57:17.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:17.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2024-04-25T07:57:17.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:17.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2024-04-25T07:57:18.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:18.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2024-04-25T07:57:18.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:18.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2024-04-25T07:57:18.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:18.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2024-04-25T07:57:19.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:19.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2024-04-25T07:57:19.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:19.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2024-04-25T07:57:20.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:20.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2024-04-25T07:57:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2024-04-25T07:57:20.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:20.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2024-04-25T07:57:21.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:21.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2024-04-25T07:57:21.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:21.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2024-04-25T07:57:22.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:22.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2024-04-25T07:57:22.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2024-04-25T07:57:22.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:22.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2024-04-25T07:57:23.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:23.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2024-04-25T07:57:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:23.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2024-04-25T07:57:23.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:23.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2024-04-25T07:57:24.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:24.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2024-04-25T07:57:24.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:24.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2024-04-25T07:57:25.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:25.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2024-04-25T07:57:25.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:25.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2024-04-25T07:57:25.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:25.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2024-04-25T07:57:26.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:26.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2024-04-25T07:57:26.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:26.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2024-04-25T07:57:26.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:26.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2024-04-25T07:57:27.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:27.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2024-04-25T07:57:27.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:27.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2024-04-25T07:57:28.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:28.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2024-04-25T07:57:28.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:28.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2024-04-25T07:57:28.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:28.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2024-04-25T07:57:29.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:29.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2024-04-25T07:57:29.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:29.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2024-04-25T07:57:29.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:29.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2024-04-25T07:57:30.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:30.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2024-04-25T07:57:30.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:30.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2024-04-25T07:57:31.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:31.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2024-04-25T07:57:31.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:31.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2024-04-25T07:57:31.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:31.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2024-04-25T07:57:32.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:32.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2024-04-25T07:57:32.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:32.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2024-04-25T07:57:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2024-04-25T07:57:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:33.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2024-04-25T07:57:33.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:33.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2024-04-25T07:57:34.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:34.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2024-04-25T07:57:34.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:34.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2024-04-25T07:57:34.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:34.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2024-04-25T07:57:35.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:35.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2024-04-25T07:57:35.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:35.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2024-04-25T07:57:35.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:35.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2024-04-25T07:57:36.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:36.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2024-04-25T07:57:36.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:36.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2024-04-25T07:57:37.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:37.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2024-04-25T07:57:37.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:37.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2024-04-25T07:57:38.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:38.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2024-04-25T07:57:38.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:38.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2024-04-25T07:57:38.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:38.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2024-04-25T07:57:39.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:39.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2024-04-25T07:57:39.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:39.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2024-04-25T07:57:39.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:39.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2024-04-25T07:57:40.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:40.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2024-04-25T07:57:40.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:40.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2024-04-25T07:57:41.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:41.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2024-04-25T07:57:41.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:41.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2024-04-25T07:57:41.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:41.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2024-04-25T07:57:42.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:42.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2024-04-25T07:57:42.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T07:57:42.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2024-04-25T07:57:42.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 3 + 1 2024-04-25T07:57:42.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=4 2024-04-25T07:57:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: get_not_primary test1 obj1-p1 2024-04-25T07:57:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-04-25T07:57:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-04-25T07:57:42.992 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-04-25T07:57:42.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-04-25T07:57:42.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-04-25T07:57:42.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T07:57:42.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T07:57:43.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T07:57:43.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T07:57:43.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T07:57:43.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: local otherosd=0 2024-04-25T07:57:43.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T07:57:44.306 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:57:44.307 INFO:tasks.workunit.client.0.smithi088.stdout:7.0 196 0 0 0 0 2048917504 0 0 586 0 586 active+clean 2024-04-25T07:52:47.455182+0000 91'586 91:664 [1] 1 [1] 1 0'0 2024-04-25T07:52:15.586094+0000 0'0 2024-04-25T07:52:15.586094+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:15:51.393769+0000 0 0 2024-04-25T07:57:44.307 INFO:tasks.workunit.client.0.smithi088.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T07:52:47.120317+0000 91'600 91:710 [1] 1 [1] 1 0'0 2024-04-25T07:52:05.141446+0000 0'0 2024-04-25T07:52:05.141446+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:53:41.694351+0000 0 0 2024-04-25T07:57:44.307 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T07:52:03.380981+0000 91'600 91:724 [1,0] 1 [1,0] 1 0'0 2024-04-25T07:52:00.405703+0000 0'0 2024-04-25T07:52:00.405703+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:29:52.928491+0000 0 0 2024-04-25T07:57:44.307 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:57:44.307 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:57:44.307 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:57:44.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ERRORS=0 2024-04-25T07:57:44.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:151: TEST_recovery_priority: ceph osd set norecover 2024-04-25T07:57:44.946 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is set 2024-04-25T07:57:44.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:152: TEST_recovery_priority: ceph osd set noout 2024-04-25T07:57:45.884 INFO:tasks.workunit.client.0.smithi088.stderr:noout is set 2024-04-25T07:57:45.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:156: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-04-25T07:57:46.587 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 7 size to 2 2024-04-25T07:57:46.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:157: TEST_recovery_priority: sleep 2 2024-04-25T07:57:48.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:57:48.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:57:48.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:57:48.612 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:57:48.612 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:48.612 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:57:48.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:57:48.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T07:57:48.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:57:48.728 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:57:48.728 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:57:48.728 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:57:48.728 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:57:48.729 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:57:48.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: seq 1 10 2024-04-25T07:57:48.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T07:57:48.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T07:57:48.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T07:57:49.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 1 = 10 ']' 2024-04-25T07:57:49.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-04-25T07:57:51.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T07:57:51.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T07:57:51.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T07:57:51.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 2 = 10 ']' 2024-04-25T07:57:51.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-04-25T07:57:53.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T07:57:53.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T07:57:53.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T07:57:53.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:164: TEST_recovery_priority: break 2024-04-25T07:57:53.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: flush_pg_stats 2024-04-25T07:57:53.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:57:53.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:57:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:57:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:57:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:57:54.410 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T07:57:54.410 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T07:57:54.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:57:54.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:57:54.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:57:54.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836568 2024-04-25T07:57:54.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836568 2024-04-25T07:57:54.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568' 2024-04-25T07:57:54.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:57:54.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:57:54.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673045 2024-04-25T07:57:54.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673045 2024-04-25T07:57:54.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673045' 2024-04-25T07:57:54.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:57:54.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:57:54.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509523 2024-04-25T07:57:54.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509523 2024-04-25T07:57:54.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673045 2-64424509523' 2024-04-25T07:57:54.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:57:54.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:57:55.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346001 2024-04-25T07:57:55.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346001 2024-04-25T07:57:55.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673045 2-64424509523 3-85899346001' 2024-04-25T07:57:55.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:57:55.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T07:57:55.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182478 2024-04-25T07:57:55.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182478 2024-04-25T07:57:55.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836568 1-42949673045 2-64424509523 3-85899346001 4-107374182478' 2024-04-25T07:57:55.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:57:55.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836568 2024-04-25T07:57:55.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:57:55.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:57:55.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836568 2024-04-25T07:57:55.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:57:55.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836568 2024-04-25T07:57:55.172 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836568 2024-04-25T07:57:55.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-04-25T07:57:55.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:57:55.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836565 -lt 21474836568 2024-04-25T07:57:55.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:57:56.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:57:56.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:57:56.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836568 -lt 21474836568 2024-04-25T07:57:56.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:57:56.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673045 2024-04-25T07:57:56.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:57:56.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:57:56.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:57:56.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673045 2024-04-25T07:57:56.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673045 2024-04-25T07:57:56.998 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673045 2024-04-25T07:57:56.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-04-25T07:57:56.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:57:57.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673045 -lt 42949673045 2024-04-25T07:57:57.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:57:57.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509523 2024-04-25T07:57:57.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:57:57.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:57:57.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509523 2024-04-25T07:57:57.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:57:57.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509523 2024-04-25T07:57:57.433 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509523 2024-04-25T07:57:57.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-04-25T07:57:57.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:57:57.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509523 -lt 64424509523 2024-04-25T07:57:57.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:57:57.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346001 2024-04-25T07:57:57.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:57:57.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:57:57.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346001 2024-04-25T07:57:57.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:57:57.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346001 2024-04-25T07:57:57.909 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899346001 2024-04-25T07:57:57.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346001' 2024-04-25T07:57:57.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:57:58.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346002 -lt 85899346001 2024-04-25T07:57:58.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:57:58.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182478 2024-04-25T07:57:58.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:57:58.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T07:57:58.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182478 2024-04-25T07:57:58.345 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:57:58.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182478 2024-04-25T07:57:58.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-04-25T07:57:58.347 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182478 2024-04-25T07:57:58.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T07:57:58.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182478 -lt 107374182478 2024-04-25T07:57:58.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:57:58.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:57:58.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:57:58.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:57:58.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:58.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:57:58.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:57:58.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T07:57:58.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:57:58.918 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:57:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:57:58.918 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:57:58.919 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:57:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:57:58.920 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:57:58.920 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:57:58.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph osd out osd.0 2024-04-25T07:57:59.435 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T07:57:59.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: sleep 2 2024-04-25T07:58:01.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:177: TEST_recovery_priority: flush_pg_stats 2024-04-25T07:58:01.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:58:01.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:58:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:58:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:58:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:58:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T07:58:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T07:58:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:58:01.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:01.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:58:02.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836571 2024-04-25T07:58:02.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836571 2024-04-25T07:58:02.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571' 2024-04-25T07:58:02.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:02.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:58:02.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673049 2024-04-25T07:58:02.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673049 2024-04-25T07:58:02.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673049' 2024-04-25T07:58:02.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:02.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:58:02.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509526 2024-04-25T07:58:02.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509526 2024-04-25T07:58:02.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673049 2-64424509526' 2024-04-25T07:58:02.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:02.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:58:02.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346005 2024-04-25T07:58:02.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346005 2024-04-25T07:58:02.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673049 2-64424509526 3-85899346005' 2024-04-25T07:58:02.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:02.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T07:58:02.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182481 2024-04-25T07:58:02.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182481 2024-04-25T07:58:02.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836571 1-42949673049 2-64424509526 3-85899346005 4-107374182481' 2024-04-25T07:58:02.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:02.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836571 2024-04-25T07:58:02.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:02.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:58:02.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836571 2024-04-25T07:58:02.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:02.748 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836571 2024-04-25T07:58:02.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836571 2024-04-25T07:58:02.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-04-25T07:58:02.749 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:58:03.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 0 -lt 21474836571 2024-04-25T07:58:03.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:58:04.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:58:04.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:58:04.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836571 -lt 21474836571 2024-04-25T07:58:04.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:04.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673049 2024-04-25T07:58:04.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:04.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:58:04.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673049 2024-04-25T07:58:04.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:04.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673049 2024-04-25T07:58:04.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-04-25T07:58:04.583 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673049 2024-04-25T07:58:04.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:58:05.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673049 -lt 42949673049 2024-04-25T07:58:05.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:05.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509526 2024-04-25T07:58:05.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:05.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:58:05.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509526 2024-04-25T07:58:05.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:05.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509526 2024-04-25T07:58:05.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-04-25T07:58:05.053 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509526 2024-04-25T07:58:05.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:58:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509526 -lt 64424509526 2024-04-25T07:58:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:05.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346005 2024-04-25T07:58:05.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:05.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:58:05.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346005 2024-04-25T07:58:05.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:05.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346005 2024-04-25T07:58:05.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346005' 2024-04-25T07:58:05.492 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899346005 2024-04-25T07:58:05.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:58:05.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346005 -lt 85899346005 2024-04-25T07:58:05.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:05.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182481 2024-04-25T07:58:05.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:05.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T07:58:05.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182481 2024-04-25T07:58:05.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:05.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182481 2024-04-25T07:58:05.911 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182481 2024-04-25T07:58:05.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-04-25T07:58:05.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T07:58:06.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182482 -lt 107374182481 2024-04-25T07:58:06.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:58:06.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:58:06.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:58:06.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:58:06.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:06.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:58:06.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:58:06.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T07:58:06.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:06.435 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:06.435 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:06.435 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:06.435 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:06.435 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:06.436 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:06.437 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:06.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T07:58:06.848 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:58:06.848 INFO:tasks.workunit.client.0.smithi088.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-04-25T07:57:53.887409+0000 91'600 104:702 [1,4] 1 [1] 1 0'0 2024-04-25T07:52:15.586094+0000 0'0 2024-04-25T07:52:15.586094+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:58:06.849 INFO:tasks.workunit.client.0.smithi088.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T07:52:47.120317+0000 91'600 104:732 [1] 1 [1] 1 0'0 2024-04-25T07:52:05.141446+0000 0'0 2024-04-25T07:52:05.141446+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:53:41.694351+0000 0 0 2024-04-25T07:58:06.849 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T07:58:00.977891+0000 91'600 104:748 [1,2] 1 [1] 1 0'0 2024-04-25T07:52:00.405703+0000 0'0 2024-04-25T07:52:00.405703+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:58:06.849 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:58:06.849 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:58:06.849 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:58:06.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-04-25T07:58:07.552 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 3 size to 2 2024-04-25T07:58:07.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: sleep 2 2024-04-25T07:58:09.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: flush_pg_stats 2024-04-25T07:58:09.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:58:09.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:58:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:58:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:58:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:58:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T07:58:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T07:58:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:58:10.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:10.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:58:10.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836575 2024-04-25T07:58:10.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836575 2024-04-25T07:58:10.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836575' 2024-04-25T07:58:10.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:10.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:58:10.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673052 2024-04-25T07:58:10.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673052 2024-04-25T07:58:10.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836575 1-42949673052' 2024-04-25T07:58:10.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:10.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:58:10.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509530 2024-04-25T07:58:10.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509530 2024-04-25T07:58:10.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836575 1-42949673052 2-64424509530' 2024-04-25T07:58:10.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:10.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:58:10.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346008 2024-04-25T07:58:10.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346008 2024-04-25T07:58:10.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836575 1-42949673052 2-64424509530 3-85899346008' 2024-04-25T07:58:10.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:10.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T07:58:10.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182485 2024-04-25T07:58:10.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182485 2024-04-25T07:58:10.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836575 1-42949673052 2-64424509530 3-85899346008 4-107374182485' 2024-04-25T07:58:10.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:10.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836575 2024-04-25T07:58:10.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:10.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:58:10.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836575 2024-04-25T07:58:10.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:10.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836575 2024-04-25T07:58:10.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-04-25T07:58:10.825 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836575 2024-04-25T07:58:10.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:58:11.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836573 -lt 21474836575 2024-04-25T07:58:11.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:58:12.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:58:12.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:58:12.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836575 -lt 21474836575 2024-04-25T07:58:12.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:12.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673052 2024-04-25T07:58:12.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:12.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:58:12.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673052 2024-04-25T07:58:12.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:12.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673052 2024-04-25T07:58:12.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-04-25T07:58:12.647 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673052 2024-04-25T07:58:12.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:58:13.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673053 -lt 42949673052 2024-04-25T07:58:13.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:13.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:13.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509530 2024-04-25T07:58:13.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:58:13.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:13.056 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509530 2024-04-25T07:58:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509530 2024-04-25T07:58:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-04-25T07:58:13.058 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509530 2024-04-25T07:58:13.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:58:13.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509530 -lt 64424509530 2024-04-25T07:58:13.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:13.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346008 2024-04-25T07:58:13.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:13.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:58:13.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346008 2024-04-25T07:58:13.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:13.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346008 2024-04-25T07:58:13.479 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899346008 2024-04-25T07:58:13.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346008' 2024-04-25T07:58:13.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:58:13.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346009 -lt 85899346008 2024-04-25T07:58:13.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:13.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182485 2024-04-25T07:58:13.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:13.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T07:58:13.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182485 2024-04-25T07:58:13.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:13.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182485 2024-04-25T07:58:13.909 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182485 2024-04-25T07:58:13.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182485' 2024-04-25T07:58:13.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T07:58:14.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182485 -lt 107374182485 2024-04-25T07:58:14.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:58:14.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:58:14.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:58:14.430 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:58:14.430 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:14.430 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:58:14.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:58:14.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T07:58:14.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:14.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:14.552 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:14.555 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:14.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T07:58:14.953 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:58:14.953 INFO:tasks.workunit.client.0.smithi088.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-04-25T07:57:53.887409+0000 91'600 108:709 [1,4] 1 [1] 1 0'0 2024-04-25T07:52:15.586094+0000 0'0 2024-04-25T07:52:15.586094+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:58:14.953 INFO:tasks.workunit.client.0.smithi088.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T07:58:09.049151+0000 91'600 108:741 [1,2] 1 [1] 1 0'0 2024-04-25T07:52:05.141446+0000 0'0 2024-04-25T07:52:05.141446+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:58:14.953 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T07:58:00.977891+0000 91'600 108:755 [1,2] 1 [1] 1 0'0 2024-04-25T07:52:00.405703+0000 0'0 2024-04-25T07:52:00.405703+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:58:14.954 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T07:58:14.954 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:58:14.954 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T07:58:14.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:14.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2024-04-25T07:58:15.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: PRIO=190 2024-04-25T07:58:15.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T07:58:15.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:15.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T07:58:15.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-04-25T07:58:15.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: ITEM=7.0 2024-04-25T07:58:15.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-04-25T07:58:15.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:15.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T07:58:15.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: PRIO=255 2024-04-25T07:58:15.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:203: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T07:58:15.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: seq 1 10 2024-04-25T07:58:15.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T07:58:15.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: ceph pg force-recovery 3.0 2024-04-25T07:58:15.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T07:58:15.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:214: TEST_recovery_priority: break 2024-04-25T07:58:15.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: sleep 2 2024-04-25T07:58:17.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:58:17.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:58:17.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:58:17.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:58:17.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:17.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:58:17.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:58:17.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T07:58:17.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:17.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:224: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:17.620 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 255, 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:17.621 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:17.622 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:17.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:17.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-04-25T07:58:17.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: PRIO=255 2024-04-25T07:58:17.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T07:58:17.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:231: TEST_recovery_priority: flush_pg_stats 2024-04-25T07:58:17.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:58:17.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:58:18.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:58:18.074 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:58:18.074 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:58:18.074 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T07:58:18.074 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T07:58:18.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:58:18.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:18.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:58:18.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836578 2024-04-25T07:58:18.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836578 2024-04-25T07:58:18.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578' 2024-04-25T07:58:18.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:18.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:58:18.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673056 2024-04-25T07:58:18.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673056 2024-04-25T07:58:18.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578 1-42949673056' 2024-04-25T07:58:18.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:18.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:58:18.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509533 2024-04-25T07:58:18.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509533 2024-04-25T07:58:18.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509533' 2024-04-25T07:58:18.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:18.523 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:58:18.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346012 2024-04-25T07:58:18.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346012 2024-04-25T07:58:18.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509533 3-85899346012' 2024-04-25T07:58:18.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:18.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T07:58:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182488 2024-04-25T07:58:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182488 2024-04-25T07:58:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509533 3-85899346012 4-107374182488' 2024-04-25T07:58:18.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:18.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836578 2024-04-25T07:58:18.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:18.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:58:18.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836578 2024-04-25T07:58:18.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:18.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836578 2024-04-25T07:58:18.815 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836578 2024-04-25T07:58:18.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-04-25T07:58:18.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:58:19.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836576 -lt 21474836578 2024-04-25T07:58:19.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:58:20.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:58:20.203 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:58:20.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836579 -lt 21474836578 2024-04-25T07:58:20.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:20.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673056 2024-04-25T07:58:20.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:20.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:58:20.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673056 2024-04-25T07:58:20.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:20.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673056 2024-04-25T07:58:20.624 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673056 2024-04-25T07:58:20.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-04-25T07:58:20.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:58:21.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673056 -lt 42949673056 2024-04-25T07:58:21.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:21.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509533 2024-04-25T07:58:21.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:21.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:58:21.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509533 2024-04-25T07:58:21.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:21.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509533 2024-04-25T07:58:21.039 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509533 2024-04-25T07:58:21.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-04-25T07:58:21.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:58:21.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509534 -lt 64424509533 2024-04-25T07:58:21.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:21.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346012 2024-04-25T07:58:21.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:21.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:58:21.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346012 2024-04-25T07:58:21.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:21.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346012 2024-04-25T07:58:21.483 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899346012 2024-04-25T07:58:21.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346012' 2024-04-25T07:58:21.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:58:21.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346013 -lt 85899346012 2024-04-25T07:58:21.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:21.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182488 2024-04-25T07:58:21.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:21.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T07:58:21.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182488 2024-04-25T07:58:21.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182488 2024-04-25T07:58:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182488' 2024-04-25T07:58:21.950 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182488 2024-04-25T07:58:21.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T07:58:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182489 -lt 107374182488 2024-04-25T07:58:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2024-04-25T07:58:22.807 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2024-04-25T07:58:22.807 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T07:58:22.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: sleep 2 2024-04-25T07:58:24.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:58:24.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:58:24.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:58:24.826 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:58:24.826 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:24.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:58:24.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:58:24.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T07:58:24.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:24.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:239: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:24.958 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:24.959 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:24.960 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:24.960 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:24.960 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:24.960 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:24.960 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:24.960 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:24.961 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:24.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:24.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2024-04-25T07:58:24.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: PRIO=190 2024-04-25T07:58:24.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T07:58:24.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:24.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T07:58:25.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: eval 'ITEM="3.0"' 2024-04-25T07:58:25.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: ITEM=3.0 2024-04-25T07:58:25.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2024-04-25T07:58:25.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:25.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T07:58:25.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: PRIO=255 2024-04-25T07:58:25.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:254: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T07:58:25.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:261: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2024-04-25T07:58:25.459 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2024-04-25T07:58:25.460 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T07:58:25.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:262: TEST_recovery_priority: sleep 5 2024-04-25T07:58:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:58:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:58:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:58:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:58:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:58:30.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:58:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T07:58:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:30.596 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:30.596 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:30.596 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:30.596 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:30.596 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:30.596 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:30.596 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:30.596 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:30.597 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:30.598 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:30.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:266: TEST_recovery_priority: flush_pg_stats 2024-04-25T07:58:30.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:58:30.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:58:31.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:58:31.016 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:58:31.016 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:58:31.016 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T07:58:31.016 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T07:58:31.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:58:31.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:31.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:58:31.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836583 2024-04-25T07:58:31.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836583 2024-04-25T07:58:31.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836583' 2024-04-25T07:58:31.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:31.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:58:31.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673061 2024-04-25T07:58:31.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673061 2024-04-25T07:58:31.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836583 1-42949673061' 2024-04-25T07:58:31.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:31.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:58:31.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509538 2024-04-25T07:58:31.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509538 2024-04-25T07:58:31.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836583 1-42949673061 2-64424509538' 2024-04-25T07:58:31.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:31.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:58:31.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346017 2024-04-25T07:58:31.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346017 2024-04-25T07:58:31.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836583 1-42949673061 2-64424509538 3-85899346017' 2024-04-25T07:58:31.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:31.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T07:58:31.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182493 2024-04-25T07:58:31.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182493 2024-04-25T07:58:31.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836583 1-42949673061 2-64424509538 3-85899346017 4-107374182493' 2024-04-25T07:58:31.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:31.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836583 2024-04-25T07:58:31.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:31.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:58:31.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836583 2024-04-25T07:58:31.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:31.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836583 2024-04-25T07:58:31.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-04-25T07:58:31.830 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836583 2024-04-25T07:58:31.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:58:32.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836583 -lt 21474836583 2024-04-25T07:58:32.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:32.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673061 2024-04-25T07:58:32.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:32.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:58:32.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673061 2024-04-25T07:58:32.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:32.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673061 2024-04-25T07:58:32.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-04-25T07:58:32.231 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673061 2024-04-25T07:58:32.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:58:32.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673061 -lt 42949673061 2024-04-25T07:58:32.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:32.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:32.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509538 2024-04-25T07:58:32.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:58:32.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509538 2024-04-25T07:58:32.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:32.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509538 2024-04-25T07:58:32.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2024-04-25T07:58:32.721 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509538 2024-04-25T07:58:32.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:58:33.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509538 -lt 64424509538 2024-04-25T07:58:33.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:33.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346017 2024-04-25T07:58:33.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:33.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:58:33.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346017 2024-04-25T07:58:33.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346017 2024-04-25T07:58:33.245 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899346017 2024-04-25T07:58:33.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346017' 2024-04-25T07:58:33.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:58:33.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346017 -lt 85899346017 2024-04-25T07:58:33.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:33.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182493 2024-04-25T07:58:33.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:33.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T07:58:33.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182493 2024-04-25T07:58:33.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:33.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182493 2024-04-25T07:58:33.712 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182493 2024-04-25T07:58:33.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2024-04-25T07:58:33.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T07:58:34.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182493 -lt 107374182493 2024-04-25T07:58:34.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T07:58:34.584 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2024-04-25T07:58:34.585 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-04-25T07:58:34.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: sleep 2 2024-04-25T07:58:36.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:58:36.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:58:36.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:58:36.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:58:36.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:36.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:58:36.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:58:36.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T07:58:36.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:36.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:271: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:36.730 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:36.730 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:36.730 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:36.730 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:36.730 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:36.731 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:36.732 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:36.733 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:36.733 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:36.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:36.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-04-25T07:58:36.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: PRIO=190 2024-04-25T07:58:36.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T07:58:36.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:36.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T07:58:36.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-04-25T07:58:36.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: ITEM=7.0 2024-04-25T07:58:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-04-25T07:58:36.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T07:58:36.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T07:58:36.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: PRIO=255 2024-04-25T07:58:36.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:286: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T07:58:36.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: ceph osd unset noout 2024-04-25T07:58:37.467 INFO:tasks.workunit.client.0.smithi088.stderr:noout is unset 2024-04-25T07:58:37.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:294: TEST_recovery_priority: ceph osd unset norecover 2024-04-25T07:58:38.246 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is unset 2024-04-25T07:58:38.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T07:58:38.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:58:38.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:58:38.266 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:58:38.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:38.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T07:58:38.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T07:58:38.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations' 2024-04-25T07:58:38.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations' 2024-04-25T07:58:38.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:58:38.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:58:38.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:58:38.268 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:58:38.268 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:58:38.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:58:38.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:58:38.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:58:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:58:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:58:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:58:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:58:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:58:38.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:58:38.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:58:38.877 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T07:58:38.877 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T07:58:38.877 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T07:58:38.877 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T07:58:38.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:58:38.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:38.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:58:39.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836587 2024-04-25T07:58:39.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836587 2024-04-25T07:58:39.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587' 2024-04-25T07:58:39.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:39.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:58:39.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673064 2024-04-25T07:58:39.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673064 2024-04-25T07:58:39.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673064' 2024-04-25T07:58:39.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:39.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:58:39.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509542 2024-04-25T07:58:39.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509542 2024-04-25T07:58:39.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673064 2-64424509542' 2024-04-25T07:58:39.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:39.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:58:39.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346020 2024-04-25T07:58:39.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346020 2024-04-25T07:58:39.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673064 2-64424509542 3-85899346020' 2024-04-25T07:58:39.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:58:39.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T07:58:39.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182497 2024-04-25T07:58:39.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182497 2024-04-25T07:58:39.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836587 1-42949673064 2-64424509542 3-85899346020 4-107374182497' 2024-04-25T07:58:39.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:39.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:39.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836587 2024-04-25T07:58:39.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:58:39.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836587 2024-04-25T07:58:39.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:39.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836587 2024-04-25T07:58:39.841 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836587 2024-04-25T07:58:39.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-04-25T07:58:39.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:58:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836587 -lt 21474836587 2024-04-25T07:58:40.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:40.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673064 2024-04-25T07:58:40.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:40.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:58:40.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673064 2024-04-25T07:58:40.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:40.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673064 2024-04-25T07:58:40.266 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673064 2024-04-25T07:58:40.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-04-25T07:58:40.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:58:40.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673064 -lt 42949673064 2024-04-25T07:58:40.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:40.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509542 2024-04-25T07:58:40.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:40.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:58:40.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509542 2024-04-25T07:58:40.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:40.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509542 2024-04-25T07:58:40.753 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509542 2024-04-25T07:58:40.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-04-25T07:58:40.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:58:41.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509542 -lt 64424509542 2024-04-25T07:58:41.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:41.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346020 2024-04-25T07:58:41.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:41.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:58:41.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346020 2024-04-25T07:58:41.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:41.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346020 2024-04-25T07:58:41.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346020' 2024-04-25T07:58:41.260 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899346020 2024-04-25T07:58:41.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:58:41.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346020 -lt 85899346020 2024-04-25T07:58:41.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:58:41.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:58:41.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182497 2024-04-25T07:58:41.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T07:58:41.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182497 2024-04-25T07:58:41.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:58:41.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182497 2024-04-25T07:58:41.761 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182497 2024-04-25T07:58:41.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182497' 2024-04-25T07:58:41.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T07:58:42.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182497 -lt 107374182497 2024-04-25T07:58:42.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:58:42.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:42.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:42.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T07:58:42.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:42.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:42.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:42.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:42.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:42.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:42.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:43.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:43.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:43.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:43.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:58:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:58:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:58:43.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:43.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:43.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:43.928 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:43.928 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:43.929 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:43.930 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:43.931 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:43.931 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:43.931 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:43.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:44.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:44.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:44.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:44.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:44.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:44.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:44.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:44.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:44.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:44.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:44.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:44.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:45.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:58:45.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8476183 2024-04-25T07:58:45.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8476183 '!=' null 2024-04-25T07:58:45.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:58:45.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:45.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:45.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:45.848 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:45.849 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:45.850 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:45.851 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:45.851 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:45.851 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:45.851 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:45.851 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:45.851 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:45.851 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:45.851 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:45.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:45.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:45.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:45.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:45.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:45.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:45.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:45.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:45.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:46.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:46.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:46.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:46.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:47.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:47.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:58:47.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6552325 2024-04-25T07:58:47.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6552325 '!=' null 2024-04-25T07:58:47.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:58:47.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:47.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:47.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:47.753 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:47.753 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:47.753 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:47.754 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:47.755 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:47.756 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:47.756 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:47.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:47.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:47.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:47.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:47.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:47.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:47.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:47.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:47.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:48.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:48.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:48.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:48.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:48.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:58:49.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16384255 2024-04-25T07:58:49.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16384255 '!=' null 2024-04-25T07:58:49.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:58:49.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:49.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:49.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:49.580 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:49.581 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:49.582 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:49.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:49.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:49.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:49.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:49.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:49.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:49.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:49.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:49.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:50.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:50.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:50.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:50.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:50.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:58:51.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13787219 2024-04-25T07:58:51.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13787219 '!=' null 2024-04-25T07:58:51.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:58:51.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:51.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:51.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:51.479 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:51.479 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:51.479 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:51.479 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:51.480 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:51.481 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:51.482 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:51.482 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:51.482 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:51.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:51.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:51.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:51.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:51.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:51.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:51.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:51.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:51.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:52.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:52.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:52.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:52.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:52.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:52.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:52.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:52.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:52.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:52.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:52.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:52.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:52.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:58:53.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21840309 2024-04-25T07:58:53.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21840309 '!=' null 2024-04-25T07:58:53.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:58:53.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:53.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:53.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:53.263 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:53.263 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:53.263 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:53.263 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:53.263 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:53.263 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:53.263 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:53.264 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:53.265 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:53.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:53.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:53.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:53.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:53.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:53.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:53.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:53.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:53.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:53.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:53.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:54.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:54.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:54.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:58:54.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471716 2024-04-25T07:58:54.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471716 '!=' null 2024-04-25T07:58:54.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:58:54.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:54.997 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:54.997 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:55.108 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:55.109 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:55.110 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:55.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:55.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:55.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:55.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:55.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:55.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:55.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:55.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:55.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:55.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:55.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:55.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:55.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:56.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:58:56.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471553 2024-04-25T07:58:56.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471553 '!=' null 2024-04-25T07:58:56.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:58:56.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:56.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:56.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:56.843 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:56.843 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:56.843 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:56.843 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:56.843 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:56.843 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:56.843 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:56.843 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:56.844 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:56.845 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:56.846 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:56.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:56.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:56.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:56.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:56.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:56.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:56.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:56.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:56.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:57.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:57.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:57.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:57.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:57.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:57.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:57.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:57.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:57.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:57.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:57.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:57.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:57.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:58:58.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080689 2024-04-25T07:58:58.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080689 '!=' null 2024-04-25T07:58:58.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:58:58.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:58.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:58:58.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:58:58.559 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:58.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:58:58.561 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:58:58.562 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:58:58.562 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:58:58.562 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:58:58.562 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:58:58.562 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:58:58.562 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:58:58.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:58:58.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:58:58.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:58:58.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:58:58.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:58:58.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:58:58.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:58:58.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:58:58.675 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:58:59.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:58:59.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:58:59.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:58:59.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:58:59.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:58:59.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:58:59.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:58:59.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:58:59.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:58:59.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:58:59.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:58:59.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:58:59.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344762 2024-04-25T07:59:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344762 '!=' null 2024-04-25T07:59:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:00.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:00.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:00.525 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:00.526 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:00.527 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:00.528 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:00.528 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:00.528 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:00.528 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:00.528 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:00.528 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:00.528 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:00.528 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:00.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:00.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:00.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:00.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:00.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:00.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:00.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:00.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:00.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:01.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:01.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:01.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:01.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:01.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:01.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:01.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:02.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27079813 2024-04-25T07:59:02.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27079813 '!=' null 2024-04-25T07:59:02.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:02.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:02.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:02.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:02.350 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:02.350 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:02.351 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:02.352 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:02.353 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:02.353 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:02.353 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:02.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:02.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:02.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:02.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:02.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:02.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:02.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:02.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:02.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:02.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:02.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:02.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:02.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:03.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:03.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:03.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:04.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344632 2024-04-25T07:59:04.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344632 '!=' null 2024-04-25T07:59:04.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:04.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:04.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:04.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:04.264 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:04.264 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:04.264 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:04.264 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:04.264 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:04.264 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:04.265 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:04.266 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:04.267 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:04.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:04.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:04.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:04.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:04.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:04.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:04.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:04.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:04.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:04.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:04.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:04.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:04.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:05.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:05.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:05.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:06.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344600 2024-04-25T07:59:06.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344600 '!=' null 2024-04-25T07:59:06.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:06.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:06.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:06.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:06.169 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:06.170 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:06.171 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:06.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:06.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:06.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:06.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:06.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:06.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:06.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:06.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:06.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:06.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:06.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:06.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:06.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:07.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:07.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:07.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:07.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:07.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:07.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:07.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:07.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:07.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:08.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27079806 2024-04-25T07:59:08.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27079806 '!=' null 2024-04-25T07:59:08.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:08.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:08.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:08.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:08.183 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:08.183 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:08.183 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:08.183 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:08.183 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:08.183 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:08.183 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:08.183 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:08.184 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:08.185 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:08.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:08.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:08.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:08.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:08.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:08.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:08.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:08.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:08.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:08.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:08.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:08.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:08.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:09.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:09.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:09.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:09.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470921 2024-04-25T07:59:09.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470921 '!=' null 2024-04-25T07:59:09.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:09.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:09.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:09.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:10.062 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:10.063 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:10.065 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:10.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:10.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:10.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:10.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:10.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:10.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:10.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:10.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:10.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:10.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:10.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:10.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:10.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:11.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:11.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:11.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:11.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:11.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:11.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:11.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:11.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:11.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:11.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470921 2024-04-25T07:59:11.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470921 '!=' null 2024-04-25T07:59:11.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:11.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:11.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:11.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:11.814 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:11.815 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:11.816 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:11.817 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:11.817 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:11.817 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:11.817 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:11.817 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:11.817 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:11.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:11.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:11.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:11.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:11.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:11.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:11.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:11.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:11.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:12.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:12.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:12.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:12.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:12.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:12.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:12.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:12.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:12.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:12.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:12.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:12.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:12.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:13.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25332928 2024-04-25T07:59:13.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25332928 '!=' null 2024-04-25T07:59:13.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:13.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:13.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:13.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:13.641 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:13.641 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:13.641 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:13.642 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:13.643 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:13.644 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:13.644 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:13.644 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:13.644 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:13.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:13.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:13.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:13.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:13.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:13.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:13.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:13.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:13.757 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:14.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:14.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:14.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:14.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:14.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:14.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:14.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:14.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:14.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:14.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:14.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:14.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:14.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:15.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597389 2024-04-25T07:59:15.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597389 '!=' null 2024-04-25T07:59:15.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:15.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:15.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:15.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:15.429 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:15.430 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:15.431 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:15.432 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:15.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:15.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:15.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:15.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:15.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:15.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:15.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:15.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:15.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:15.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:15.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:15.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:15.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:16.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:17.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597228 2024-04-25T07:59:17.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597228 '!=' null 2024-04-25T07:59:17.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:17.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:17.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:17.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:17.157 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:17.158 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:17.160 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:17.160 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:17.160 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:17.160 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:17.160 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:17.160 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:17.160 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:17.160 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:17.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:17.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:17.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:17.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:17.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:17.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:17.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:17.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:17.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:17.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:17.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:17.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:17.772 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:18.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:18.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:19.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333476 2024-04-25T07:59:19.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333476 '!=' null 2024-04-25T07:59:19.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:19.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:19.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:19.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:19.142 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:19.143 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:19.144 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:19.145 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:19.145 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:19.145 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:19.145 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:19.145 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:19.145 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:19.145 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:19.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:19.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:19.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:19.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:19.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:19.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:19.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:19.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:19.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:19.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:19.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:19.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:19.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:20.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:20.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:20.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:20.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:20.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:20.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:20.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:20.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:20.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:20.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598138 2024-04-25T07:59:20.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598138 '!=' null 2024-04-25T07:59:20.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:20.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:20.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:20.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:20.970 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:20.970 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:20.970 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:20.970 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:20.970 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:20.971 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:20.972 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:20.973 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:20.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:21.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:21.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:21.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:21.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:21.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:21.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:21.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:21.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:21.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:21.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:21.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:21.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:22.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:22.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:22.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:22.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:22.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:22.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333995 2024-04-25T07:59:22.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333995 '!=' null 2024-04-25T07:59:22.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:22.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:22.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:22.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:22.778 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:22.779 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:22.780 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:22.781 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:22.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:22.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:22.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:22.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:22.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:22.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:22.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:22.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:22.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:23.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:23.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:23.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:23.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:23.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:23.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:23.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:23.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:23.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:23.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:23.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:23.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:23.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:24.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471829 2024-04-25T07:59:24.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471829 '!=' null 2024-04-25T07:59:24.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:24.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:24.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:24.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:24.732 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:24.732 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:24.732 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:24.732 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:24.732 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:24.732 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:24.732 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:24.733 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:24.734 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:24.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:24.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:24.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:24.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:24.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:24.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:24.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:25.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:25.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:25.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:25.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:25.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:25.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:25.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:26.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472047 2024-04-25T07:59:26.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472047 '!=' null 2024-04-25T07:59:26.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:26.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:26.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:26.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:26.720 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:26.720 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:26.720 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:26.720 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:26.720 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:26.720 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:26.720 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:26.723 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:26.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:26.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:26.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:26.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:26.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:26.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:26.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:26.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:26.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:27.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:27.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:27.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:27.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:28.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:28.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:28.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:28.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:28.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:28.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:28.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:28.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:28.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:28.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207919 2024-04-25T07:59:28.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207919 '!=' null 2024-04-25T07:59:28.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:28.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:28.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:28.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:28.809 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:28.809 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:28.809 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:28.810 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:28.811 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:28.812 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:28.812 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:28.812 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:28.812 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:28.812 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:28.812 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:28.812 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:28.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:28.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:28.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:28.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:28.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:28.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:28.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:28.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:28.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:29.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:29.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:29.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:29.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:29.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:29.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:29.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:29.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:29.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:29.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:29.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:29.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:29.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:30.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471483 2024-04-25T07:59:30.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471483 '!=' null 2024-04-25T07:59:30.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:30.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:30.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:30.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:30.614 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:30.615 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:30.616 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:30.617 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:30.617 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:30.617 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:30.617 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:30.617 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:30.617 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:30.617 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:30.617 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:30.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:30.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:30.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:30.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:30.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:30.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:30.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:30.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:30.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:31.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:31.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:31.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:31.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:31.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:31.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:31.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:31.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:31.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:31.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:31.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:31.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:31.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:32.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207072 2024-04-25T07:59:32.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207072 '!=' null 2024-04-25T07:59:32.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:32.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:32.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:32.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:32.580 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:32.580 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:32.580 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:32.580 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:32.580 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:32.581 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:32.582 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:32.583 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:32.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:32.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:32.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:32.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:32.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:32.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:32.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:32.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:32.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:33.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:33.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:33.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:33.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:33.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:33.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:33.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:33.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:33.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:33.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:33.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:33.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:33.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:34.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471557 2024-04-25T07:59:34.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471557 '!=' null 2024-04-25T07:59:34.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:34.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:34.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:34.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:34.437 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:34.437 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:34.437 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:34.437 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:34.437 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:34.437 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:34.438 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:34.439 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:34.440 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:34.440 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:34.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:34.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:34.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:34.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:34.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:34.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:34.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:34.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:34.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:34.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:34.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:34.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:34.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:35.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:35.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:35.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:35.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:35.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:35.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:35.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:35.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:35.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:36.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471445 2024-04-25T07:59:36.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471445 '!=' null 2024-04-25T07:59:36.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:36.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:36.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:36.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:36.372 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:36.373 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:36.374 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:36.375 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:36.375 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:36.375 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:36.375 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:36.375 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:36.375 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:36.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:36.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:36.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:36.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:36.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:36.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:36.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:36.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:36.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:36.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:36.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:36.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:36.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:37.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:37.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:37.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:37.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:37.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:37.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:37.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:37.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:37.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:38.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333701 2024-04-25T07:59:38.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333701 '!=' null 2024-04-25T07:59:38.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:38.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:38.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:38.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:38.273 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:38.273 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:38.274 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:38.275 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:38.276 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:38.276 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:38.276 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:38.276 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:38.276 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:38.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:38.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:38.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:38.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:38.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:38.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:38.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:38.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:38.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:38.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:38.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:38.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:38.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:39.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:39.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:39.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:39.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:39.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:39.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:39.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:39.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:39.512 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:40.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598247 2024-04-25T07:59:40.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598247 '!=' null 2024-04-25T07:59:40.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:40.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:40.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:40.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:40.374 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:40.374 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:40.375 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:40.376 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:40.377 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:40.377 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:40.377 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:40.377 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:40.377 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:40.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:40.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:40.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:40.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:40.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:40.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:40.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:41.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:41.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:41.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:41.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:41.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:41.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:41.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:41.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:41.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:42.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25334297 2024-04-25T07:59:42.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25334297 '!=' null 2024-04-25T07:59:42.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:42.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:42.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:42.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:42.226 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:42.227 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:42.228 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:42.228 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:42.228 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:42.228 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:42.228 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:42.228 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:42.228 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:42.228 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:42.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:42.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:42.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:42.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:42.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:42.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:42.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:42.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:42.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:42.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:42.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:42.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:42.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:43.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:43.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:43.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:43.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:43.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:43.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:43.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:43.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:43.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:43.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25334297 2024-04-25T07:59:43.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25334297 '!=' null 2024-04-25T07:59:43.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:43.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:43.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:43.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:43.953 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:43.953 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:43.953 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:43.954 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:43.955 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:43.956 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:43.956 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:43.956 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:43.956 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:43.956 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:43.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:44.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:44.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:44.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:44.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:44.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:44.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:44.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:44.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:44.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:44.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:44.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:44.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:45.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:45.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:45.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:45.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597981 2024-04-25T07:59:45.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597981 '!=' null 2024-04-25T07:59:45.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:45.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:45.656 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:45.656 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:45.760 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:45.760 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:45.760 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:45.760 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:45.760 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:45.760 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:45.760 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:45.761 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:45.762 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:45.763 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:45.763 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:45.763 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:45.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:45.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:45.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:45.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:45.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:45.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:45.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:45.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:45.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:46.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:46.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:46.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:46.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:46.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:46.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:46.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:47.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597760 2024-04-25T07:59:47.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597760 '!=' null 2024-04-25T07:59:47.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:47.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:47.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:47.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:47.631 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:47.632 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:47.633 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:47.634 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:47.634 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:47.634 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:47.634 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:47.634 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:47.634 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:47.634 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:47.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:47.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:47.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:47.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:47.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:47.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:47.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:47.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:47.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:48.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:48.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:48.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:48.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:48.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:48.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:48.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:48.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:48.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:48.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:48.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:48.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:48.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:49.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333435 2024-04-25T07:59:49.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333435 '!=' null 2024-04-25T07:59:49.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:49.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:49.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:49.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:49.696 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:49.697 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:49.698 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:49.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:49.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:49.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:49.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:49.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:49.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:49.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:49.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:49.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:50.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:50.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:50.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:50.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:50.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:50.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:50.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:50.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:50.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:50.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:50.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:50.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:51.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471053 2024-04-25T07:59:51.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471053 '!=' null 2024-04-25T07:59:51.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:51.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:51.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:51.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:51.650 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:51.651 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:51.652 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:51.653 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:51.653 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:51.653 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:51.653 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:51.653 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:51.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:51.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:51.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:51.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:51.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:51.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:51.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:51.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:51.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:52.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:52.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:52.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:52.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:52.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:52.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:52.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206149 2024-04-25T07:59:53.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206149 '!=' null 2024-04-25T07:59:53.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:53.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:53.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:53.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:53.565 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:53.566 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:53.567 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:53.568 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:53.568 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:53.568 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:53.568 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:53.568 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:53.568 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:53.568 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:53.568 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:53.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:53.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:53.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:53.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:53.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:53.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:54.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:54.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:54.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:54.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:54.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:55.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470669 2024-04-25T07:59:55.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470669 '!=' null 2024-04-25T07:59:55.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:55.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:55.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:55.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:55.387 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:55.388 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:55.389 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:55.390 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:55.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:55.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:55.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:55.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:55.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:55.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:55.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:55.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:55.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:55.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:55.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:55.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:55.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:56.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:56.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:56.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:56.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:56.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:56.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:56.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:56.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:56.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:57.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470798 2024-04-25T07:59:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470798 '!=' null 2024-04-25T07:59:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:57.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:57.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:57.247 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:57.249 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:57.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:57.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:57.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:57.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:57.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:57.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:57.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:57.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:57.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:57.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:57.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:57.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:57.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:59:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T07:59:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:59:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:59:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:59:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:59:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:59:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:59:58.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:59:58.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:59:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26205959 2024-04-25T07:59:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26205959 '!=' null 2024-04-25T07:59:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T07:59:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:58.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T07:59:58.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:59.054 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T07:59:59.055 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T07:59:59.056 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T07:59:59.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:59:59.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:59:59.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:59:59.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:59:59.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:59:59.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:59:59.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:59:59.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:59:59.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:59:59.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:59:59.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:59:59.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:59:59.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:00.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:00.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470755 2024-04-25T08:00:00.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470755 '!=' null 2024-04-25T08:00:00.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:00.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:00.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:00.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:00.806 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:00.806 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:00.806 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:00.806 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:00.806 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:00.807 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:00.808 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:00.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:00.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:00.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:00.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:00.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:00.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:00.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:01.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:01.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:01.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:01.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:02.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:02.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:02.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206746 2024-04-25T08:00:02.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206746 '!=' null 2024-04-25T08:00:02.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:02.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:02.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:02.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:02.822 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:02.822 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:02.822 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:02.823 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:02.824 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:02.825 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:02.825 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:02.825 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:02.825 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:02.825 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:02.825 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:02.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:02.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:02.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:02.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:02.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:02.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:02.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:03.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:03.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:03.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:03.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:03.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:03.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:03.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:03.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:03.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:03.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:03.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:03.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:03.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:04.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471356 2024-04-25T08:00:04.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471356 '!=' null 2024-04-25T08:00:04.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:04.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:04.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:04.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:04.618 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:04.618 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:04.618 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:04.618 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:04.619 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:04.620 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:04.621 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:04.621 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:04.621 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:04.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:04.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:04.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:04.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:04.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:04.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:04.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:04.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:04.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:05.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:05.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:05.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:05.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:06.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:06.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:06.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:06.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:06.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:06.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:06.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:06.036 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:06.036 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:06.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471397 2024-04-25T08:00:06.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471397 '!=' null 2024-04-25T08:00:06.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:06.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:06.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:06.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:06.790 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:06.790 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:06.790 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:06.790 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:06.791 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:06.792 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:06.793 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:06.793 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:06.793 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:06.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:06.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:06.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:06.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:06.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:06.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:06.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:06.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:06.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:07.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:07.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:07.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:07.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:08.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:08.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:08.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:08.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:08.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:08.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:08.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:08.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:08.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:08.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207128 2024-04-25T08:00:08.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207128 '!=' null 2024-04-25T08:00:08.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:08.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:08.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:08.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:08.709 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:08.709 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:08.709 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:08.709 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:08.709 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:08.709 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:08.710 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:08.711 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:08.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:08.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:08.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:08.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:08.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:08.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:08.824 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:08.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:08.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:09.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:09.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:09.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:09.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:09.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:10.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471791 2024-04-25T08:00:10.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471791 '!=' null 2024-04-25T08:00:10.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:10.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:10.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:10.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:10.441 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:10.442 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:10.443 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:10.443 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:10.443 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:10.443 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:10.443 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:10.443 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:10.443 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:10.443 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:10.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:10.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:10.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:10.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:10.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:10.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:10.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:10.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:10.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:10.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:10.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:10.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:10.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:11.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:11.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:12.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207661 2024-04-25T08:00:12.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207661 '!=' null 2024-04-25T08:00:12.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:12.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:12.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:12.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:12.323 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:12.324 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:12.325 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:12.326 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:12.326 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:12.326 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:12.326 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:12.326 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:12.326 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:12.327 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:12.327 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:12.327 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:12.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:12.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:12.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:12.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:12.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:12.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:12.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:12.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:12.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:12.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:12.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:12.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:13.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:14.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471728 2024-04-25T08:00:14.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471728 '!=' null 2024-04-25T08:00:14.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:14.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:14.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:14.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:14.117 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:14.118 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:14.119 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:14.120 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:14.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:14.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:14.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:14.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:14.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:14.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:14.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:14.235 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:14.235 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:14.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:14.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:14.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:14.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:15.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:15.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:15.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:15.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:15.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:15.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:15.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:15.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:15.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:16.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471550 2024-04-25T08:00:16.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471550 '!=' null 2024-04-25T08:00:16.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:16.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:16.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:16.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:16.108 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:16.109 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:16.110 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:16.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:16.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:16.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:16.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:16.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:16.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:16.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:16.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:16.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:16.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:16.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:16.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:16.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:17.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:17.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:17.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:17.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:17.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:17.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:17.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:17.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:17.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:17.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471550 2024-04-25T08:00:17.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471550 '!=' null 2024-04-25T08:00:17.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:17.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:17.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:17.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:17.874 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:17.874 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:17.875 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "7.0", 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 255, 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:17.876 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:17.877 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:17.877 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:17.877 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:17.877 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:17.877 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:17.877 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:17.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:17.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:17.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:17.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:17.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:17.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:17.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:17.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:17.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:18.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:18.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:18.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:18.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:19.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:19.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:19.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:19.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:19.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:19.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:19.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:19.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:19.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:19.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207035 2024-04-25T08:00:19.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207035 '!=' null 2024-04-25T08:00:19.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:19.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:19.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:19.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:19.923 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:19.924 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:19.925 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:19.925 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:19.925 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:19.925 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:19.925 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:19.925 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:19.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:20.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:20.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:20.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:20.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:20.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:20.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:20.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:20.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:20.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:00:20.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:20.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:20.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:21.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:00:21.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:00:21.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:21.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:21.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:21.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:21.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:21.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:21.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:21.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471270 2024-04-25T08:00:21.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471270 '!=' null 2024-04-25T08:00:21.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:21.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:21.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:21.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:21.978 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:21.979 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:21.980 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:21.980 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:21.980 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:21.980 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:21.980 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:21.980 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:21.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:22.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:22.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:22.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:22.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:22.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:22.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:22.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:22.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:22.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:22.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:23.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:23.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-04-25T08:00:23.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:00:23.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T08:00:23.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:23.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:23.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:23.188 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:23.189 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:23.190 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:23.190 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:23.190 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:23.190 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:23.190 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:23.190 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:23.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:23.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:23.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:23.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:23.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:23.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:23.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:23.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:23.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:23.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:23.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:23.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:23.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:24.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:24.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:24.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:24.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:24.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:24.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:24.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:24.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:24.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:25.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24895618 2024-04-25T08:00:25.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24895618 '!=' null 2024-04-25T08:00:25.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:25.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:25.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:25.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:25.124 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:25.125 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:25.126 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:25.126 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:25.126 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:25.126 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:25.126 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:25.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:25.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:25.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:25.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:25.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:25.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:25.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:25.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:25.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:25.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:25.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:25.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:25.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:26.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:26.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:26.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:26.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:26.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:26.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:26.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:26.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:26.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:26.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793225 2024-04-25T08:00:26.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793225 '!=' null 2024-04-25T08:00:26.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:26.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:26.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:26.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:27.039 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:27.040 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:27.041 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:27.041 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:27.041 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:27.041 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:27.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:27.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:27.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:27.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:27.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:27.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:27.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:27.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:27.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:27.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:27.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:27.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:27.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:28.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:28.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:28.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:28.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:28.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:28.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:28.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:28.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:28.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:28.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24897539 2024-04-25T08:00:28.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24897539 '!=' null 2024-04-25T08:00:28.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:28.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:28.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:28.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:28.806 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:28.807 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:28.808 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:28.808 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:28.808 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:28.808 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:28.808 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:28.808 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:28.808 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:28.808 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:28.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:28.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:28.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:28.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:28.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:28.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:28.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:28.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:28.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:29.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:29.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:29.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:29.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:29.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:29.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:29.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:30.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19944352 2024-04-25T08:00:30.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19944352 '!=' null 2024-04-25T08:00:30.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:30.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:30.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:30.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:30.751 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:30.751 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:30.751 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:30.751 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:30.751 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:30.751 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:30.751 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:30.752 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:30.753 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:30.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:30.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:30.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:30.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:30.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:30.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:30.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:30.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:30.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:31.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:31.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:31.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:31.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:32.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:32.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:32.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:32.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:32.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:32.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:32.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:32.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:32.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:32.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26842638 2024-04-25T08:00:32.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26842638 '!=' null 2024-04-25T08:00:32.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:32.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:32.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:32.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:32.722 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:32.722 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:32.722 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:32.722 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:32.722 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:32.723 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:32.724 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:32.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:32.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:32.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:32.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:32.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:32.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:32.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:32.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:32.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:33.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:33.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:33.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:33.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:33.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:33.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:33.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:34.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471966 2024-04-25T08:00:34.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471966 '!=' null 2024-04-25T08:00:34.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:34.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:34.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:34.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:34.653 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:34.654 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:34.655 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:34.655 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:34.655 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:34.655 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:34.655 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:34.655 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:34.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:34.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:34.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:34.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:34.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:34.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:34.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:34.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:34.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:35.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:35.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:35.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:35.195 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:35.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:35.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:35.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:35.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:35.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:35.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:35.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:35.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:35.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:36.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471876 2024-04-25T08:00:36.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471876 '!=' null 2024-04-25T08:00:36.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:36.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:36.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:36.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:36.537 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:36.537 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:36.537 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:36.537 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:36.537 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:36.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:36.537 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:36.538 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:36.539 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:36.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:36.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:36.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:36.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:36.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:36.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:36.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:36.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:36.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:37.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:37.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:37.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:37.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:37.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:37.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:37.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:38.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207487 2024-04-25T08:00:38.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207487 '!=' null 2024-04-25T08:00:38.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:38.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:38.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:38.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:38.388 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:38.388 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:38.388 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:38.388 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:38.388 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:38.388 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:38.388 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:38.388 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:38.389 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:38.390 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:38.390 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:38.390 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:38.390 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:38.390 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:38.390 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:38.390 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:38.390 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:38.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:38.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:38.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:38.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:38.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:38.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:38.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:38.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:38.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:38.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:38.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:38.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:38.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:39.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:39.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:39.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:39.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:39.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:39.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:39.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:39.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:39.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:40.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471663 2024-04-25T08:00:40.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471663 '!=' null 2024-04-25T08:00:40.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:40.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:40.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:40.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:40.250 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:40.250 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:40.250 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:40.250 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:40.250 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:40.250 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:40.250 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:40.250 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:40.251 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:40.252 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:40.252 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:40.252 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:40.252 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:40.252 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:40.252 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:40.252 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:40.252 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:40.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:40.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:40.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:40.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:40.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:40.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:40.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:40.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:40.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:40.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:40.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:40.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:40.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:41.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:41.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:41.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:42.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206557 2024-04-25T08:00:42.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206557 '!=' null 2024-04-25T08:00:42.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:42.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:42.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:42.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:42.114 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:42.115 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:42.116 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:42.116 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:42.116 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:42.116 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:42.116 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:42.116 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:42.116 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:42.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:42.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:42.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:42.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:42.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:42.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:42.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:42.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:42.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:42.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:42.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:42.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:42.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:43.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:43.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:43.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:43.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:43.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:43.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:43.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:43.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:43.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:43.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206557 2024-04-25T08:00:43.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206557 '!=' null 2024-04-25T08:00:43.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:43.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:43.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:43.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:44.094 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:44.094 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:44.094 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:44.094 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:44.094 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:44.094 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:44.096 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:44.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:44.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:44.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:44.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:44.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:44.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:45.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:45.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:45.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:45.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:45.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:45.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:45.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:45.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:45.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:45.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470940 2024-04-25T08:00:45.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470940 '!=' null 2024-04-25T08:00:45.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:45.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:45.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:45.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:45.900 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:45.901 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:45.902 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:45.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:46.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:46.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:46.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:46.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:46.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:46.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:46.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:46.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:46.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:46.436 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:46.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:46.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:46.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:46.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:46.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:46.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:46.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:46.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:46.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:46.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:46.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:47.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470720 2024-04-25T08:00:47.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470720 '!=' null 2024-04-25T08:00:47.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:47.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:47.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:47.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:47.776 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:47.777 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:47.778 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:47.778 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:47.778 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:47.778 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:47.778 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:47.778 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:47.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:47.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:47.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:47.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:47.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:47.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:47.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:47.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:47.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:48.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:48.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:48.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:48.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:48.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:48.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:48.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:48.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:48.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:48.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:48.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:48.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:48.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:49.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206197 2024-04-25T08:00:49.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206197 '!=' null 2024-04-25T08:00:49.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:49.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:49.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:49.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:49.772 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:49.772 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:49.772 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:49.772 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:49.772 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:49.773 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:49.774 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:49.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:49.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:49.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:49.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:49.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:49.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:49.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:49.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:49.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:50.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:50.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:50.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:50.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:51.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:51.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:51.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:51.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:51.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:51.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:51.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:51.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:51.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:51.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470946 2024-04-25T08:00:51.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470946 '!=' null 2024-04-25T08:00:51.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:51.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:51.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:51.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:51.894 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:51.895 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:51.896 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:51.896 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:51.896 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:51.896 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:51.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:52.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:52.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:52.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:52.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:52.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:52.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:52.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:52.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:52.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:52.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:52.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:52.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:52.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:52.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:52.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:52.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:52.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:52.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:52.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:52.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:52.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206331 2024-04-25T08:00:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206331 '!=' null 2024-04-25T08:00:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:53.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:53.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:53.671 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:53.671 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:53.671 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:53.671 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:53.671 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:53.671 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:53.671 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:53.672 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:53.673 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:53.673 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:53.673 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:53.673 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:53.673 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:53.673 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:53.673 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:53.673 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:53.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:53.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:53.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:53.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:53.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:53.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:53.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:53.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:53.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:54.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:54.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:54.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:54.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:54.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:54.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:54.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:55.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471011 2024-04-25T08:00:55.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471011 '!=' null 2024-04-25T08:00:55.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:55.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:55.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:55.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:55.434 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:55.434 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:55.434 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:55.434 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:55.434 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:55.434 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:55.435 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:55.436 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:55.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:55.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:55.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:55.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:55.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:55.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:55.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:55.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:55.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:55.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:55.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:55.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:55.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:56.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:56.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:56.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:56.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:56.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:56.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:56.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:56.549 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:56.549 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:57.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471087 2024-04-25T08:00:57.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471087 '!=' null 2024-04-25T08:00:57.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:57.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:57.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:57.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:57.268 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:57.268 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:57.268 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:57.268 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:57.269 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:57.270 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:57.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:57.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:57.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:57.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:57.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:57.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:57.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:57.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:57.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:57.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:57.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:57.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:57.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:00:58.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:00:58.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207356 2024-04-25T08:00:58.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207356 '!=' null 2024-04-25T08:00:58.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:00:58.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:58.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:00:58.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:00:59.050 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:00:59.050 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:00:59.050 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:59.050 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:59.050 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:00:59.050 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:59.050 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:00:59.051 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:00:59.052 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:00:59.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:00:59.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:00:59.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:00:59.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:00:59.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:00:59.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:00:59.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:00:59.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:00:59.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:00:59.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:00:59.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:00:59.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:00:59.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:00.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:00.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:00.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:00.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:00.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:00.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:00.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:00.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:00.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:00.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471517 2024-04-25T08:01:00.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471517 '!=' null 2024-04-25T08:01:00.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:00.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:00.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:00.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:00.795 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:00.795 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:00.795 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:00.795 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:00.795 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:00.795 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:00.795 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:00.795 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:00.796 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:00.797 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:00.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:00.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:00.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:00.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:00.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:00.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:00.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:00.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:00.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:01.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:01.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:01.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:01.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:01.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:01.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:01.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:01.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:01.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:01.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:01.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:01.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:01.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:02.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207418 2024-04-25T08:01:02.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207418 '!=' null 2024-04-25T08:01:02.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:02.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:02.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:02.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:02.606 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:02.607 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:02.608 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:02.609 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:02.609 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:02.609 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:02.609 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:02.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:02.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:02.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:02.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:02.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:02.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:02.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:02.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:02.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:03.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:03.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:03.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:03.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:03.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:03.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:03.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:03.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:04.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471352 2024-04-25T08:01:04.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471352 '!=' null 2024-04-25T08:01:04.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:04.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:04.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:04.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:04.357 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:04.358 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:04.359 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:04.359 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:04.359 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:04.359 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:04.359 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:04.359 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:04.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:04.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:04.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:04.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:04.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:04.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:04.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:04.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:04.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:04.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:04.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:04.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:04.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:05.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:05.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:05.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:05.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:05.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:05.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:05.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:05.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:05.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:06.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471775 2024-04-25T08:01:06.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471775 '!=' null 2024-04-25T08:01:06.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:06.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:06.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:06.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:06.183 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:06.184 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:06.185 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:06.185 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:06.185 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:06.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:06.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:06.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:06.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:06.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:06.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:06.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:06.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:06.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:06.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:06.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:06.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:06.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:07.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:07.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:07.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:07.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:07.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:07.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:07.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:07.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:07.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:07.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471775 2024-04-25T08:01:07.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471775 '!=' null 2024-04-25T08:01:07.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:07.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:07.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:07.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:08.078 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:08.078 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:08.079 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:08.080 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:08.081 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:08.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:08.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:08.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:08.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:08.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:08.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:08.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:08.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:08.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:08.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:08.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:08.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:08.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:09.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:09.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:09.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:09.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:09.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:09.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:09.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:09.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:09.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:09.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207292 2024-04-25T08:01:09.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207292 '!=' null 2024-04-25T08:01:09.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:09.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:09.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:09.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:09.799 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:09.799 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:09.799 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:09.799 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:09.799 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:09.799 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:09.799 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:09.799 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:09.800 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:09.801 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:09.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:09.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:09.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:09.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:09.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:09.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:09.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:09.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:09.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:10.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:10.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:10.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:10.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:10.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:10.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:10.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:10.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:10.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:10.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:10.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:10.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:11.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471090 2024-04-25T08:01:11.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471090 '!=' null 2024-04-25T08:01:11.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:11.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:11.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:11.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:11.594 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:11.594 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:11.594 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:11.594 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:11.595 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:11.596 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:11.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:11.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:11.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:11.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:11.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:11.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:11.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:11.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:11.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:12.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:12.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:12.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:12.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:12.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:12.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:12.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:12.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:12.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:12.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:12.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:12.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:12.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:13.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206892 2024-04-25T08:01:13.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206892 '!=' null 2024-04-25T08:01:13.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:13.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:13.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:13.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:13.590 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:13.591 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:13.592 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:13.593 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:13.593 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:13.593 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:13.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:13.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:13.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:13.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:13.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:13.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:13.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:13.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:13.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:14.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:14.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:14.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:14.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:14.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:14.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:14.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:14.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:14.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:14.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:14.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:14.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:14.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:15.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470983 2024-04-25T08:01:15.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470983 '!=' null 2024-04-25T08:01:15.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:15.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:15.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:15.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:15.676 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:15.676 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:15.676 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:15.676 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:15.676 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:15.676 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:15.676 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:15.677 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:15.678 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:15.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:15.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:15.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:15.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:15.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:15.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:15.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:15.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:15.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:16.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:16.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:16.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:16.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:16.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:16.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:16.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:16.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:16.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:17.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470977 2024-04-25T08:01:17.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470977 '!=' null 2024-04-25T08:01:17.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:17.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:17.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:17.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:17.605 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:17.605 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:17.605 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:17.606 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:17.607 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:17.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:17.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:17.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:17.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:17.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:17.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:17.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:17.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:17.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:18.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:18.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:18.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:18.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:18.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:18.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:18.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:18.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:18.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:18.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:18.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:18.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:18.747 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:19.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206388 2024-04-25T08:01:19.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206388 '!=' null 2024-04-25T08:01:19.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:19.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:19.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:19.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:19.410 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:19.411 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:19.412 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:19.412 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:19.412 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:19.412 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:19.412 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:19.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:19.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:19.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:19.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:19.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:19.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:19.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:19.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:20.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:20.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:20.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:20.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470792 2024-04-25T08:01:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470792 '!=' null 2024-04-25T08:01:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:21.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:21.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:21.189 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:21.189 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:21.189 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:21.190 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:21.191 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:21.192 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:21.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:21.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:21.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:21.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:21.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:21.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:21.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:21.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:21.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:21.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:21.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:21.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:21.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:22.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:22.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:22.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:22.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:22.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:22.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:22.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:22.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:22.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:23.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080228 2024-04-25T08:01:23.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080228 '!=' null 2024-04-25T08:01:23.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:23.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:23.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:23.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:23.112 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:23.112 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:23.112 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:23.112 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:23.112 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:23.113 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:23.114 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:23.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:23.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:23.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:23.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:23.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:23.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:23.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:23.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:23.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:23.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:23.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:23.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:23.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:24.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:24.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:24.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:24.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:24.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:24.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:24.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:24.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:24.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:24.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344840 2024-04-25T08:01:24.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344840 '!=' null 2024-04-25T08:01:24.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:24.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:24.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:24.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:24.939 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:24.939 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:24.939 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:24.939 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:24.939 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:24.939 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:24.940 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:24.941 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:24.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:25.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:25.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:25.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:25.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:25.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:25.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:25.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:25.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:25.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:25.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:25.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:26.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:26.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:26.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:26.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:26.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:26.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:26.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:26.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:26.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:26.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18345075 2024-04-25T08:01:26.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18345075 '!=' null 2024-04-25T08:01:26.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:26.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:26.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:26.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:26.721 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:26.722 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:26.723 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:26.723 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:26.723 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:26.723 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:26.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:26.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:26.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:26.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:26.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:26.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:26.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:26.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:26.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:27.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:27.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:27.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:27.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:27.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:27.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:27.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:27.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:27.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:27.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:28.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080859 2024-04-25T08:01:28.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080859 '!=' null 2024-04-25T08:01:28.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:28.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:28.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:28.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:28.404 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:28.404 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:28.404 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:28.404 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:28.404 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:28.404 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:28.404 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:28.405 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:28.406 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:28.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:28.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:28.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:28.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:28.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:28.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:28.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:28.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:28.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:28.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:28.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:28.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:28.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:29.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:29.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:29.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:29.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:29.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:29.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:29.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:29.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:29.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:30.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344962 2024-04-25T08:01:30.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344962 '!=' null 2024-04-25T08:01:30.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:30.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:30.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:30.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:30.196 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:30.196 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:30.196 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:30.196 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:30.196 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:30.196 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:30.196 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:30.197 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:30.198 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:30.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:30.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:30.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:30.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:30.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:30.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:30.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:30.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:30.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:30.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:30.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:30.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:30.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:31.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:31.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344962 2024-04-25T08:01:31.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344962 '!=' null 2024-04-25T08:01:31.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:31.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:31.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:31.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:31.913 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:31.914 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:31.915 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:31.915 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:31.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:32.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:32.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:32.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:32.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:32.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:32.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:32.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:32.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:32.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:32.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:32.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:32.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:33.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:33.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27081510 2024-04-25T08:01:33.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27081510 '!=' null 2024-04-25T08:01:33.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:33.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:33.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:33.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:33.734 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:33.734 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:33.734 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:33.734 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:33.735 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:33.736 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:33.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:33.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:33.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:33.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:33.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:33.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:33.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:33.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:33.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:34.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:34.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:34.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:34.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:34.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:34.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:34.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:35.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471850 2024-04-25T08:01:35.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471850 '!=' null 2024-04-25T08:01:35.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:35.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:35.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:35.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:35.457 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:35.458 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:35.459 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:35.460 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:35.460 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:35.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:35.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:35.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:35.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:35.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:35.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:35.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:35.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:35.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:35.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:35.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:35.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:36.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:36.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:36.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:36.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:36.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:36.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:36.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:36.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:36.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:37.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471911 2024-04-25T08:01:37.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471911 '!=' null 2024-04-25T08:01:37.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:37.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:37.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:37.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:37.199 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:37.200 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:37.201 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:37.202 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:37.202 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:37.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:37.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:37.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:37.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:37.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:37.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:37.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:37.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:37.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:37.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:37.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:37.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:37.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:38.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:38.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:38.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:38.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:38.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:38.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:38.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:38.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:38.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:39.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207691 2024-04-25T08:01:39.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207691 '!=' null 2024-04-25T08:01:39.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:39.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:39.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:39.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:39.329 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:39.329 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:39.330 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:39.331 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:39.332 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:39.332 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:39.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:39.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:39.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:39.444 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:39.444 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:39.444 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:39.444 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:39.444 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:39.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:39.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:39.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:39.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:39.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:40.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:40.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:40.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:40.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:40.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:40.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:40.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:40.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:40.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:41.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471765 2024-04-25T08:01:41.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471765 '!=' null 2024-04-25T08:01:41.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:41.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:41.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:41.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:41.252 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:41.252 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:41.252 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:41.252 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:41.252 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:41.252 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:41.255 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:41.255 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:41.255 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:41.255 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:41.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:41.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:41.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:41.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:41.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:41.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:41.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:41.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:41.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:41.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:41.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:41.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:41.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:42.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:42.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333886 2024-04-25T08:01:42.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333886 '!=' null 2024-04-25T08:01:42.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:42.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:42.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:42.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:43.020 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:43.020 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:43.020 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:43.020 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:43.020 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:43.020 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:43.021 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:43.022 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:43.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:43.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:43.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:43.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:43.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:43.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:43.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:43.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:43.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:43.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:43.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:43.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:43.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:44.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:44.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:44.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:44.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:44.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:44.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:44.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:44.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:44.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:44.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597676 2024-04-25T08:01:44.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597676 '!=' null 2024-04-25T08:01:44.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:44.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:44.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:44.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:44.910 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:44.911 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:44.912 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:44.913 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:44.913 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:44.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:45.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:45.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:45.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:45.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:45.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:45.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:45.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:45.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:45.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:45.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:45.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:45.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:46.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:46.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:46.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:46.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:46.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:46.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:46.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:46.116 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:46.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:46.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597645 2024-04-25T08:01:46.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597645 '!=' null 2024-04-25T08:01:46.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:46.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:46.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:46.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:46.804 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:46.804 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:46.805 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:46.806 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:46.807 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:46.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:46.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:46.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:46.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:46.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:46.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:46.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:46.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:46.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:47.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:47.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:47.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:47.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:47.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:47.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:47.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:47.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:47.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:47.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:47.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:47.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:47.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:48.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585454 2024-04-25T08:01:48.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585454 '!=' null 2024-04-25T08:01:48.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:48.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:48.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:48.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:48.616 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:48.617 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:48.618 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:48.619 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:48.619 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:48.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:48.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:48.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:48.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:48.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:48.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:48.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:48.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:48.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:49.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:49.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:49.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:49.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:49.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:49.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:49.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:49.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:49.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:49.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:49.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:49.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:49.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:50.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850134 2024-04-25T08:01:50.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850134 '!=' null 2024-04-25T08:01:50.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:50.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:50.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:50.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:50.537 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:50.537 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:50.537 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:50.538 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:50.539 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:50.540 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:50.540 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:50.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:50.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:50.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:50.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:50.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:50.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:50.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:50.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:50.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:51.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:51.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:51.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:51.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:51.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:51.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:51.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:51.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:51.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:51.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:51.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:51.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:51.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:52.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838606 2024-04-25T08:01:52.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838606 '!=' null 2024-04-25T08:01:52.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:52.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:52.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:52.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:52.379 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:52.379 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:52.379 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:52.379 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:52.379 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:52.380 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:52.381 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:52.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:52.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:52.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:52.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:52.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:52.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:52.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:52.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:52.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:52.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:52.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:52.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:52.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:53.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:53.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:53.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:54.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976561 2024-04-25T08:01:54.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976561 '!=' null 2024-04-25T08:01:54.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:54.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:54.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:54.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:54.372 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:54.372 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:54.373 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:54.374 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:54.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:54.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:54.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:54.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:54.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:54.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:54.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:54.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:54.488 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:55.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:55.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:55.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:55.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:55.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:55.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:55.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:55.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:55.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:55.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:55.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:55.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:55.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:56.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976769 2024-04-25T08:01:56.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976769 '!=' null 2024-04-25T08:01:56.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:56.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:56.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:56.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:56.441 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:56.441 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:56.441 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:56.441 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:56.441 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:56.441 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:56.441 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:56.442 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:56.443 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:56.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:56.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:56.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:56.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:56.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:56.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:56.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:56.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:56.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:57.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:57.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:57.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:57.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:57.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:01:58.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965245 2024-04-25T08:01:58.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965245 '!=' null 2024-04-25T08:01:58.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:01:58.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:58.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:01:58.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:01:58.437 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:01:58.437 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:58.438 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:01:58.439 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:01:58.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:01:58.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:01:58.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:01:58.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:01:58.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:01:58.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:01:58.553 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:01:58.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:01:58.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:01:59.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:01:59.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:01:59.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:01:59.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:01:59.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:01:59.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:01:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:01:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:01:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:01:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:01:59.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:01:59.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:01:59.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:00.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977055 2024-04-25T08:02:00.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977055 '!=' null 2024-04-25T08:02:00.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:00.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:00.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:00.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:00.218 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:00.218 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:00.218 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:00.218 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:00.218 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:02:00.218 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:00.218 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:02:00.218 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:00.219 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:00.220 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:00.220 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:00.220 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:00.220 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:00.220 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:00.220 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:00.220 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:00.220 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:00.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:00.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:00.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:00.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:00.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:00.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:00.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:00.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:00.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:00.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:02:00.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:00.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:00.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:02:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:02:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:01.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:01.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:01.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:01.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977055 2024-04-25T08:02:01.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977055 '!=' null 2024-04-25T08:02:01.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:01.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:01.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:01.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:02:01.996 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:01.997 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:01.998 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:01.998 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:01.998 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:01.998 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:01.998 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:01.998 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:01.998 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:01.998 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:02.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:02.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:02.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:02.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:02.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:02.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:02.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:02.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:02.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:02.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:02:02.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:02.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:02.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:03.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:03.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965743 2024-04-25T08:02:03.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965743 '!=' null 2024-04-25T08:02:03.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:03.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:03.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:03.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:03.847 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:03.847 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [ 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 190, 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:02:03.848 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "1.0", 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:03.849 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:03.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:03.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:03.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:03.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:03.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:03.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:03.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:03.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:03.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:04.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:02:04.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:04.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:04.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:05.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:02:05.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:02:05.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:05.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:05.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:05.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:05.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:05.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:05.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:05.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977273 2024-04-25T08:02:05.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977273 '!=' null 2024-04-25T08:02:05.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:05.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:05.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:05.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:05.976 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:05.976 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:05.977 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:05.978 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:05.978 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:05.978 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:05.978 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:05.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:06.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:06.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:06.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:06.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:06.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:06.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:06.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:06.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:06.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:02:06.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:06.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:06.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:02:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:02:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:07.575 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:07.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:08.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19656252 2024-04-25T08:02:08.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19656252 '!=' null 2024-04-25T08:02:08.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:08.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:08.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:08.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:08.291 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:08.291 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:08.291 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:08.291 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:08.291 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:08.291 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:08.292 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:08.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:08.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:08.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:08.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:08.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:08.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:08.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:08.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:08.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:09.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:02:09.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:09.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:09.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:09.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:02:09.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:02:09.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:09.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:09.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:09.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:09.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:09.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:09.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19656252 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19656252 '!=' null 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:10.401 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:10.401 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:10.401 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:10.401 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:10.402 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:10.403 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:10.403 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:10.403 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:10.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:10.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:10.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:10.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:10.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:10.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:10.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:10.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:10.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:11.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:02:11.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:11.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:11.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:02:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:02:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:11.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:11.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:11.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:12.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966598 2024-04-25T08:02:12.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966598 '!=' null 2024-04-25T08:02:12.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:12.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:12.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:12.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:12.434 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:12.435 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:12.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:12.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:12.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:12.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:12.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:12.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:12.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:12.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:12.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:12.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:12.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:12.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:12.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:13.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:13.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 1 2024-04-25T08:02:13.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:02:13.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-04-25T08:02:13.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:13.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:13.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:13.670 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:13.671 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:13.672 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:13.672 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:13.672 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:13.672 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:13.672 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:13.672 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:13.672 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:13.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:13.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:13.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:13.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:13.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:13.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:13.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:13.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:13.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:14.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:14.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:14.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:14.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:15.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:15.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:15.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:15.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:15.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:15.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:15.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:15.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:15.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18692710 2024-04-25T08:02:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18692710 '!=' null 2024-04-25T08:02:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:15.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:15.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:15.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:15.981 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:15.982 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:15.982 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:15.982 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:15.982 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:15.982 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:15.982 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:15.982 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:15.983 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:15.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:16.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:16.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:16.096 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:16.096 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:16.096 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:16.096 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:16.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:16.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:16.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:16.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:16.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:16.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:17.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:17.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:17.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:17.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:17.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:17.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:17.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:17.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:17.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:17.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16772805 2024-04-25T08:02:17.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16772805 '!=' null 2024-04-25T08:02:17.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:17.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:17.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:17.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:17.840 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:17.841 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:17.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:17.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:17.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:17.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:17.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:17.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:17.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:17.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:17.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:18.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:18.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:18.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:18.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:19.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:19.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:19.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:19.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:19.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:19.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:19.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:19.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:19.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:19.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21021307 2024-04-25T08:02:19.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21021307 '!=' null 2024-04-25T08:02:19.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:19.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:19.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:19.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:19.702 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:19.702 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:19.702 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:19.702 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:19.702 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:19.702 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:19.703 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:19.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:19.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:19.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:19.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:19.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:19.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:19.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:19.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:19.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:20.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:20.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:20.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:20.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:20.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:21.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850519 2024-04-25T08:02:21.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850519 '!=' null 2024-04-25T08:02:21.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:21.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:21.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:21.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:21.669 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:21.670 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:21.670 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:21.670 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:21.670 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:21.670 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:21.670 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:21.670 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:21.670 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:21.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:21.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:21.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:21.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:21.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:21.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:21.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:21.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:21.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:22.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:22.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:22.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:22.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:22.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:22.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:22.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838176 2024-04-25T08:02:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838176 '!=' null 2024-04-25T08:02:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:23.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:23.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:23.914 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:23.915 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:23.915 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:23.915 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:23.915 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:23.915 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:23.915 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:23.915 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:23.915 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:23.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:24.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:24.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:24.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:24.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:24.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:24.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:24.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:24.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:24.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:24.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:24.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:24.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:25.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:25.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976160 2024-04-25T08:02:25.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976160 '!=' null 2024-04-25T08:02:25.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:25.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:25.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:25.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:25.692 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:25.693 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:25.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:25.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:25.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:25.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:25.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:25.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:25.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:25.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:25.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:26.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:26.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:26.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:26.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:26.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:26.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:26.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:26.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:26.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:26.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:26.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:26.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:26.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:27.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976070 2024-04-25T08:02:27.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976070 '!=' null 2024-04-25T08:02:27.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:27.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:27.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:27.448 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:27.555 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:27.555 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:27.555 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:27.556 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:27.557 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:27.557 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:27.557 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:27.557 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:27.557 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:27.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:27.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:27.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:27.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:27.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:27.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:27.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:28.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:28.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:28.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:28.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:28.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:28.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:28.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:28.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:28.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:28.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:28.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:28.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:28.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:29.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20964520 2024-04-25T08:02:29.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20964520 '!=' null 2024-04-25T08:02:29.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:29.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:29.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:29.194 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:29.302 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:29.303 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:29.304 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:29.304 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:29.304 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:29.304 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:29.304 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:29.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:29.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:29.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:29.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:29.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:29.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:29.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:29.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:29.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:29.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:29.924 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:29.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:29.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:30.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:31.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976361 2024-04-25T08:02:31.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976361 '!=' null 2024-04-25T08:02:31.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:31.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:31.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:31.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:31.113 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:31.113 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:31.113 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:31.113 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:31.113 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:31.113 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:31.113 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:31.114 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:31.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:31.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:31.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:31.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:31.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:31.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:31.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:31.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:31.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:31.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:31.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:31.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:31.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:32.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:32.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:32.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:32.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:32.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:32.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:32.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:32.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:32.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:32.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838000 2024-04-25T08:02:32.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838000 '!=' null 2024-04-25T08:02:32.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:32.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:32.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:32.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:32.860 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:32.861 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:32.862 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:32.862 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:32.862 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:32.862 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:32.862 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:32.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:32.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:32.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:32.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:32.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:32.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:32.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:32.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:32.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:33.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:33.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:33.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:33.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:33.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:33.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:33.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:33.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:33.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:33.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:33.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:33.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:33.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850251 2024-04-25T08:02:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850251 '!=' null 2024-04-25T08:02:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:34.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:34.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:34.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:34.721 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:34.722 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:34.723 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:34.723 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:34.723 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:34.723 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:34.723 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:34.723 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:34.723 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:34.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:34.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:34.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:34.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:34.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:34.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:34.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:34.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:34.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:35.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:35.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:35.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:35.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:35.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:36.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850583 2024-04-25T08:02:36.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850583 '!=' null 2024-04-25T08:02:36.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:36.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:36.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:36.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:36.516 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:36.516 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:36.516 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:36.516 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:36.516 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:36.517 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:36.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:36.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:36.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:36.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:36.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:36.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:36.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:36.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:36.632 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:37.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:37.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:37.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:37.097 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:37.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:37.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:37.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:37.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:37.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:37.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:37.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:37.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:37.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:38.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839338 2024-04-25T08:02:38.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839338 '!=' null 2024-04-25T08:02:38.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:38.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:38.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:38.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:38.524 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:38.524 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:38.524 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:38.524 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:38.524 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:38.525 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:38.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:38.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:38.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:38.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:38.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:38.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:38.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:39.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:39.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:39.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:39.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:39.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:39.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:39.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:39.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:39.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:39.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:39.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:39.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:39.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:40.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850462 2024-04-25T08:02:40.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850462 '!=' null 2024-04-25T08:02:40.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:40.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:40.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:40.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:40.322 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:40.322 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:40.322 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:40.323 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:40.324 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:40.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:40.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:40.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:40.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:40.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:40.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:40.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:40.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:40.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:40.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:40.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:40.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:40.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:41.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:41.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:41.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:41.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:41.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:41.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:41.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:41.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:41.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:42.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838967 2024-04-25T08:02:42.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838967 '!=' null 2024-04-25T08:02:42.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:42.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:42.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:42.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:42.400 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:42.401 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:42.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:42.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:42.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:42.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:42.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:42.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:42.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:42.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:42.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:42.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:42.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:43.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:43.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:43.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:43.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:43.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:43.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:43.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:43.749 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:43.749 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:44.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977036 2024-04-25T08:02:44.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977036 '!=' null 2024-04-25T08:02:44.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:44.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:44.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:44.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:44.467 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:44.468 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:44.468 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:44.468 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:44.468 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:44.468 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:44.468 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:44.468 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:44.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:44.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:44.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:44.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:44.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:44.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:44.583 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:44.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:44.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:44.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:44.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:44.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:44.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:45.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:45.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:45.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:45.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:45.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:45.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:45.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:45.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:45.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:46.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976789 2024-04-25T08:02:46.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976789 '!=' null 2024-04-25T08:02:46.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:46.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:46.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:46.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:46.305 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:46.306 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:46.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:46.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:46.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:46.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:46.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:46.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:46.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:46.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:46.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:46.908 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:46.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:46.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:47.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:47.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:47.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:47.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:47.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:47.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:47.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:47.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:47.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:48.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965278 2024-04-25T08:02:48.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965278 '!=' null 2024-04-25T08:02:48.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:48.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:48.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:48.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:48.403 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:48.403 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:48.403 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:48.403 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:48.403 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:48.403 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:48.404 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:48.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:48.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:48.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:48.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:48.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:48.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:48.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:48.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:48.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:48.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:48.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:48.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:48.996 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:49.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:49.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:49.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:49.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:49.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:49.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:49.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:49.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:49.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:50.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976943 2024-04-25T08:02:50.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976943 '!=' null 2024-04-25T08:02:50.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:50.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:50.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:50.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:50.338 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:50.339 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:50.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:50.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:50.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:50.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:50.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:50.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:50.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:50.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:50.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:50.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:50.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:50.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:50.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:51.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:51.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:51.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:51.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:51.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:51.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:51.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:51.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:51.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:52.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965442 2024-04-25T08:02:52.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965442 '!=' null 2024-04-25T08:02:52.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:52.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:52.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:52.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:52.195 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:52.196 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:52.196 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:52.196 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:52.196 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:52.196 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:52.196 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:52.196 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:52.196 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:52.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:52.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:52.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:52.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:52.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:52.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:52.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:52.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:52.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:52.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:52.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:52.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:52.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:53.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:53.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:54.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977012 2024-04-25T08:02:54.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977012 '!=' null 2024-04-25T08:02:54.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:54.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:54.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:54.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:54.158 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:54.158 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:54.159 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:54.160 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:54.160 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:54.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:54.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:54.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:54.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:54.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:54.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:54.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:54.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:54.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:54.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:54.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:54.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:54.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:55.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:55.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:55.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:55.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:55.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:55.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:56.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976964 2024-04-25T08:02:56.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976964 '!=' null 2024-04-25T08:02:56.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:56.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:56.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:56.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:56.211 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:56.211 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:56.211 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:56.212 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:56.213 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:56.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:56.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:56.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:56.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:56.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:56.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:56.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:56.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:56.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:56.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:56.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:56.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:56.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:57.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:57.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:57.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:57.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:57.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:57.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:57.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:57.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:57.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:02:58.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839255 2024-04-25T08:02:58.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839255 '!=' null 2024-04-25T08:02:58.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:02:58.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:58.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:02:58.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:02:58.497 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:02:58.498 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:02:58.499 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:02:58.499 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:02:58.499 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:02:58.499 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:02:58.499 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:02:58.499 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:02:58.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:02:58.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:02:58.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:02:58.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:02:58.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:02:58.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:02:58.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:02:58.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:02:58.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:02:59.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:02:59.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:02:59.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:02:59.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:02:59.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:02:59.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:02:59.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:02:59.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:02:59.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:02:59.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:02:59.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:02:59.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:02:59.766 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:00.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850446 2024-04-25T08:03:00.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850446 '!=' null 2024-04-25T08:03:00.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:00.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:00.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:00.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:00.457 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:00.458 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:00.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:00.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:00.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:00.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:00.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:00.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:00.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:00.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:00.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:00.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:00.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:00.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:00.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:01.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:01.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:01.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:01.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:01.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:01.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:01.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:01.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:01.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:02.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839044 2024-04-25T08:03:02.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839044 '!=' null 2024-04-25T08:03:02.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:02.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:02.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:02.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:02.200 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:02.200 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:02.200 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:02.201 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:02.202 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:02.202 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:02.202 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:02.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:02.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:02.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:02.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:02.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:02.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:02.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:02.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:02.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:02.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:02.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:02.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:02.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:03.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:03.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:03.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:03.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:03.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:03.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:03.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:03.447 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:03.447 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:04.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839044 2024-04-25T08:03:04.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839044 '!=' null 2024-04-25T08:03:04.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:04.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:04.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:04.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:04.097 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:04.098 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:04.099 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:04.099 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:04.099 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:04.099 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:04.099 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:04.099 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:04.099 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:04.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:04.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:04.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:04.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:04.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:04.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:04.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:04.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:04.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:04.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:04.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:04.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:05.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:05.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:05.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:05.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:05.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:05.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:05.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:05.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:05.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:05.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850579 2024-04-25T08:03:05.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850579 '!=' null 2024-04-25T08:03:05.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:05.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:05.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:05.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:05.823 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:05.824 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:05.824 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:05.824 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:05.824 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:05.824 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:05.824 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:05.824 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:05.824 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:05.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:05.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:05.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:05.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:05.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:05.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:05.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:05.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:05.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:06.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:06.393 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:06.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:06.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:06.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:06.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:06.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:06.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:06.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:06.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:06.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:06.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:06.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:07.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850834 2024-04-25T08:03:07.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850834 '!=' null 2024-04-25T08:03:07.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:07.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:07.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:07.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:07.688 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:07.688 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:07.689 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:07.690 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:07.690 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:07.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:07.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:07.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:07.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:07.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:07.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:07.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:07.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:07.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:08.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:08.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:08.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:08.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:09.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:09.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:09.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:09.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:09.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:09.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:09.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:09.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:09.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:09.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713424 2024-04-25T08:03:09.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713424 '!=' null 2024-04-25T08:03:09.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:09.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:09.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:09.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:10.064 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:10.065 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:10.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:10.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:10.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:10.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:10.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:10.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:10.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:10.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:10.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:10.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:10.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:10.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:10.597 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:11.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:11.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:11.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:11.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:11.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:11.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:11.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:11.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:11.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:11.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850995 2024-04-25T08:03:11.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850995 '!=' null 2024-04-25T08:03:11.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:11.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:11.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:11.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:11.967 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:11.968 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:11.969 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:11.969 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:11.969 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:11.969 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:11.969 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:11.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:12.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:12.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:12.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:12.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:12.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:12.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:12.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:12.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:12.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:12.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:12.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:12.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:13.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:13.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:13.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:13.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966049 2024-04-25T08:03:13.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966049 '!=' null 2024-04-25T08:03:13.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:13.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:13.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:13.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:13.817 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:13.817 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:13.817 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:13.818 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:13.819 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:13.819 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:13.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:13.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:13.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:13.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:13.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:13.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:13.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:13.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:14.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:14.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:14.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:14.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:15.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:15.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:15.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:15.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:15.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:15.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:15.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:15.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:15.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:15.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977061 2024-04-25T08:03:15.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977061 '!=' null 2024-04-25T08:03:15.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:15.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:15.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:15.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:15.840 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:15.840 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:15.840 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:15.841 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:15.842 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:15.842 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:15.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:15.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:15.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:15.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:15.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:15.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:15.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:15.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:15.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:16.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:16.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:16.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:16.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:17.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:17.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:17.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:17.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:17.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:17.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:17.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:17.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:17.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:17.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977256 2024-04-25T08:03:17.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977256 '!=' null 2024-04-25T08:03:17.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:17.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:17.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:17.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:17.726 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:17.726 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:17.726 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:17.727 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:17.728 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:17.728 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:17.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:17.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:17.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:17.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:17.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:17.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:17.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:17.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:17.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:18.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:18.443 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:18.443 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:18.443 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:18.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:18.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:18.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:18.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:18.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:18.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:18.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:18.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:18.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:19.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965818 2024-04-25T08:03:19.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965818 '!=' null 2024-04-25T08:03:19.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:19.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:19.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:19.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:19.795 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:19.795 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:19.795 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:19.795 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:19.795 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:19.796 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:19.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:19.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:19.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:19.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:19.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:19.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:19.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:19.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:19.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:20.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:20.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:20.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:20.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:21.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:21.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:21.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13103802 2024-04-25T08:03:21.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13103802 '!=' null 2024-04-25T08:03:21.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:21.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:21.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:21.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:21.927 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:21.927 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:21.927 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:21.927 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:21.927 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:21.927 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:21.927 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:21.928 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:21.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:22.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:22.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:22.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:22.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:22.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:22.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:22.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:22.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:22.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:22.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:22.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:22.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:23.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:23.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:23.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:23.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:23.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:23.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:23.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:23.122 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:23.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:24.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20092728 2024-04-25T08:03:24.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20092728 '!=' null 2024-04-25T08:03:24.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:24.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:24.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:24.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:24.161 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:24.162 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:24.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:24.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:24.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:24.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:24.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:24.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:24.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:24.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:24.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:24.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:24.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:24.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:24.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:25.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:25.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:25.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:25.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:25.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:25.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:25.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:25.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:25.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:25.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977582 2024-04-25T08:03:25.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977582 '!=' null 2024-04-25T08:03:25.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:25.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:25.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:25.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:26.089 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:26.090 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:26.091 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:26.091 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:26.091 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:26.091 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:26.091 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:26.091 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:26.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:26.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:26.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:26.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:26.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:26.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:26.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:26.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:26.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:26.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:26.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:26.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:26.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:27.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:27.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:27.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:27.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:27.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:27.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:27.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:27.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:27.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:27.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977680 2024-04-25T08:03:27.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977680 '!=' null 2024-04-25T08:03:27.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:27.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:27.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:27.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:27.837 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:27.837 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:27.837 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:27.837 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:27.837 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:27.837 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:27.838 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:27.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:27.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:27.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:27.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:27.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:27.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:27.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:27.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:27.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:28.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:28.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:28.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:28.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:28.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:28.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:28.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:28.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:28.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:28.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:28.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:28.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:28.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:29.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966001 2024-04-25T08:03:29.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966001 '!=' null 2024-04-25T08:03:29.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:29.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:29.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:29.631 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:29.734 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:29.734 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:29.734 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:29.734 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:29.734 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:29.734 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:29.735 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:29.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:29.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:29.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:29.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:29.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:29.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:29.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:29.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:29.853 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:30.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:30.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:30.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:30.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:31.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:31.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:31.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:31.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:31.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:31.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:31.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:31.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:31.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:31.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977286 2024-04-25T08:03:31.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977286 '!=' null 2024-04-25T08:03:31.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:31.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:31.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:31.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:31.727 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:31.727 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:31.727 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:31.728 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:31.729 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:31.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:31.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:31.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:31.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:31.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:31.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:31.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:31.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:31.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:32.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:32.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:32.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:33.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:33.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:33.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:33.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:33.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:33.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:33.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:33.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:33.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:33.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965518 2024-04-25T08:03:33.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965518 '!=' null 2024-04-25T08:03:33.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:33.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:33.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:33.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:33.799 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:33.799 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:33.799 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:33.799 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:33.799 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:33.800 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:33.801 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:33.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:33.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:33.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:33.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:33.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:33.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:33.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:33.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:33.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:34.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:34.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:34.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:34.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:35.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:35.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:35.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:35.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:35.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:35.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:35.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:35.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:35.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:35.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976878 2024-04-25T08:03:35.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976878 '!=' null 2024-04-25T08:03:35.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:35.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:35.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:35.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:35.755 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:35.755 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:35.755 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:35.755 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:35.755 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:35.755 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:35.755 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:35.755 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:35.756 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:35.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:35.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:35.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:35.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:35.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:35.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:35.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:35.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:35.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:36.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:36.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:36.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:36.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:36.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:36.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:36.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:36.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:36.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:36.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:36.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:37.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976957 2024-04-25T08:03:37.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976957 '!=' null 2024-04-25T08:03:37.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:37.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:37.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:37.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:37.476 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:37.477 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:37.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:37.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:37.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:37.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:37.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:37.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:37.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:37.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:37.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:38.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:38.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:38.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:38.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:38.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:38.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:38.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:38.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:38.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:38.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:38.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:38.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:38.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:39.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965481 2024-04-25T08:03:39.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965481 '!=' null 2024-04-25T08:03:39.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:39.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:39.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:39.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:39.426 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:39.426 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:39.426 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:39.426 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:39.426 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:39.426 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:39.426 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:39.426 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:39.427 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:39.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:39.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:39.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:39.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:39.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:39.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:39.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:39.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:39.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:40.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:40.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:40.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:40.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:40.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:40.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:40.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:41.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977244 2024-04-25T08:03:41.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977244 '!=' null 2024-04-25T08:03:41.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:41.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:41.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:41.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:41.253 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:41.254 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:41.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:41.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:41.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:41.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:41.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:41.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:41.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:41.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:41.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:41.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:41.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:41.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:41.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:42.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:42.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:42.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:42.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:42.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:42.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:42.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:42.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:42.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:42.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965516 2024-04-25T08:03:42.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965516 '!=' null 2024-04-25T08:03:42.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:42.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:42.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:42.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:43.092 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:43.092 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:43.092 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:43.092 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:43.092 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:43.092 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:43.092 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:43.092 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:43.093 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:43.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:43.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:43.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:43.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:43.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:43.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:43.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:43.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:43.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:43.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:43.630 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:43.630 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:43.630 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:44.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:44.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:44.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:44.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977193 2024-04-25T08:03:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977193 '!=' null 2024-04-25T08:03:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:44.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:44.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:45.063 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:45.064 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:45.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:45.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:45.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:45.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:45.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:45.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:45.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:45.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:45.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:45.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:45.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:45.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:45.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:46.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:46.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:46.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:46.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:46.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:46.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:46.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:46.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:46.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:46.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977499 2024-04-25T08:03:46.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977499 '!=' null 2024-04-25T08:03:46.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:46.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:46.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:46.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:46.970 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:46.970 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:46.971 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:46.972 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:46.972 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:46.972 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:46.972 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:46.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:47.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:47.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:47.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:47.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:47.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:47.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:47.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:47.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:47.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:47.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:47.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:47.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:48.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:48.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:48.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:48.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:48.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:48.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:48.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:48.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:48.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:48.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966452 2024-04-25T08:03:48.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966452 '!=' null 2024-04-25T08:03:48.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:48.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:48.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:48.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:48.755 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:48.755 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:48.756 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:48.757 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:48.757 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:48.757 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:48.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:48.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:48.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:48.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:48.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:48.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:48.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:48.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:48.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:49.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:49.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:49.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:49.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:49.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:49.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:49.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:49.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:49.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:49.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:49.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:49.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:49.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:50.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977561 2024-04-25T08:03:50.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977561 '!=' null 2024-04-25T08:03:50.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:50.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:50.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:50.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:50.626 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:50.626 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:50.626 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:50.626 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:50.627 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:50.627 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:50.627 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:50.627 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:50.627 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:50.627 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:50.628 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:50.628 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:50.628 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:50.628 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:50.628 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:50.628 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:50.629 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:50.629 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:50.629 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:50.629 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:50.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:50.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:50.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:50.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:50.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:50.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:50.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:50.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:50.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:51.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:51.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:51.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:51.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:51.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:51.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:51.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:51.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:51.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:51.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:51.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:51.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:51.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:52.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966249 2024-04-25T08:03:52.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966249 '!=' null 2024-04-25T08:03:52.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:52.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:52.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:52.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:52.430 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:52.430 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:52.430 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:52.430 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:52.430 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:52.431 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:52.432 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:52.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:52.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:52.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:52.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:52.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:52.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:52.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:52.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:52.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:52.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:52.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:52.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:52.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:53.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:53.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:53.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:53.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:53.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:53.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:53.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:53.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:53.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:54.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966249 2024-04-25T08:03:54.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966249 '!=' null 2024-04-25T08:03:54.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:54.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:54.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:54.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:54.205 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:54.206 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:54.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:54.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:54.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:54.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:54.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:54.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:54.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:54.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:54.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:54.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:54.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:54.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:54.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:55.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:55.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:55.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:55.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:55.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:55.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:55.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:55.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:55.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:56.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977638 2024-04-25T08:03:56.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977638 '!=' null 2024-04-25T08:03:56.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:56.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:56.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:56.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:56.157 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:56.157 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:56.157 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:56.157 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:56.157 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:56.157 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:56.157 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:56.157 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:56.158 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:56.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:56.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:56.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:56.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:56.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:56.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:56.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:56.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:56.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:56.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:56.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:56.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:56.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:57.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:57.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:57.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:57.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:57.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:57.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:57.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:57.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:57.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:57.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977638 2024-04-25T08:03:57.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977638 '!=' null 2024-04-25T08:03:57.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:57.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:57.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:57.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:58.026 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:58.026 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:58.027 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:58.028 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:58.028 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:58.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:03:58.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:03:58.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:03:58.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:03:58.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:03:58.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:03:58.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:03:58.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:03:58.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:03:58.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:03:58.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:03:58.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:03:58.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:03:59.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:03:59.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:03:59.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:03:59.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:03:59.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:03:59.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:03:59.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:03:59.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:03:59.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:03:59.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965718 2024-04-25T08:03:59.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965718 '!=' null 2024-04-25T08:03:59.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:03:59.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:59.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:03:59.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:03:59.933 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:03:59.934 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:03:59.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:00.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:00.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:00.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:00.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:00.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:00.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:00.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:04:00.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:00.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:00.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:04:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:04:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:01.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:01.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976682 2024-04-25T08:04:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976682 '!=' null 2024-04-25T08:04:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:01.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:04:01.661 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:04:01.765 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:01.766 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:04:01.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:01.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:01.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:01.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:01.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:01.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:01.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:01.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:01.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:02.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:04:02.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:02.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:02.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:04:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:04:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:02.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:02.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:02.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:03.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965180 2024-04-25T08:04:03.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965180 '!=' null 2024-04-25T08:04:03.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:03.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:03.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:04:03.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:03.727 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:04:03.727 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:03.728 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:03.729 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:03.729 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:04:03.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:03.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:03.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:03.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:03.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:03.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:03.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:03.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:03.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:04.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:04:04.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:04.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:04.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:04.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:04:04.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:04:04.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:04.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:04.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:04.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:04.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:04.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:04.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:05.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976894 2024-04-25T08:04:05.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976894 '!=' null 2024-04-25T08:04:05.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:05.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:05.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:04:05.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:05.625 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:04:05.625 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:04:05.625 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:05.625 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:05.625 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:05.625 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:04:05.625 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:05.625 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:05.626 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:04:05.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:05.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:05.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:05.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:05.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:05.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:05.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:05.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:05.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:06.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:04:06.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:06.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:06.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:04:06.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:04:06.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:06.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:06.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:06.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:06.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:06.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:06.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:07.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976942 2024-04-25T08:04:07.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976942 '!=' null 2024-04-25T08:04:07.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:07.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:07.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:04:07.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:07.461 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [ 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: "item": "3.0", 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: "prio": 190, 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: "can_preempt": true 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:07.462 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:07.463 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:04:07.463 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:07.463 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:07.463 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:07.463 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:07.463 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:07.463 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:04:07.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:07.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:07.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:07.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:07.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:07.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:07.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:07.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:04:07.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:07.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:07.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:08.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:04:08.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:04:08.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:08.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:08.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:08.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:08.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:08.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:08.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:09.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839010 2024-04-25T08:04:09.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839010 '!=' null 2024-04-25T08:04:09.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:09.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:09.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:04:09.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:09.343 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:04:09.343 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:04:09.343 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:09.344 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:04:09.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:09.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:09.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:09.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:09.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:09.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:09.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:09.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:09.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:09.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:04:09.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:09.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:09.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:10.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:10.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:10.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793648 2024-04-25T08:04:10.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793648 '!=' null 2024-04-25T08:04:10.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:10.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:10.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:04:10.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:11.090 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:11.091 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:11.091 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:11.091 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:04:11.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:11.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:11.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:11.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:11.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:11.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:11.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:11.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:11.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:11.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:04:11.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:11.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:11.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:04:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:04:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:04:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:04:12.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:04:12.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:04:12.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:04:12.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:04:12.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:04:12.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14414484 2024-04-25T08:04:12.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14414484 '!=' null 2024-04-25T08:04:12.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:04:12.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:12.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:04:12.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:04:12.954 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "local_reservations": { 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "remote_reservations": { 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "max_allowed": 1, 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "min_priority": 0, 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "queues": [], 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: "in_progress": [] 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:12.955 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:04:12.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:04:13.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:04:13.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:13.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:13.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:13.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:13.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:13.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:13.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:13.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:04:13.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:13.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:13.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:13.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:04:13.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:04:13.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:04:13.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:298: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:04:14.396 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:04:14.396 INFO:tasks.workunit.client.0.smithi088.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:00:20.729266+0000 91'600 118:1141 [1,4] 1 [1,4] 1 0'0 2024-04-25T07:52:15.586094+0000 0'0 2024-04-25T07:52:15.586094+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:53:12.227189+0000 0 0 2024-04-25T08:04:14.396 INFO:tasks.workunit.client.0.smithi088.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:04:08.741190+0000 91'600 118:1174 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:52:05.141446+0000 0'0 2024-04-25T07:52:05.141446+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:13:04.586881+0000 0 0 2024-04-25T08:04:14.396 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:02:06.874703+0000 91'600 118:1183 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:52:00.405703+0000 0'0 2024-04-25T07:52:00.405703+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:02:53.051331+0000 0 0 2024-04-25T08:04:14.396 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:04:14.396 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:04:14.397 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T08:04:14.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:04:14.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:04:14.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:04:14.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:04:14.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:14.414 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T08:04:14.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202479/ceph-osd.1.asok 2024-04-25T08:04:14.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:04:14.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.202479/ceph-osd.1.asok dump_pgstate_history 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: "pgs": [ 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: "pg": "7.0", 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: "history": [ 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "60", 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.520 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:15.757996+0000", 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:15.758006+0000" 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:15.758012+0000", 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:15.892156+0000" 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:15.892159+0000", 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:15.892393+0000" 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.521 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:15.892394+0000", 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:15.892439+0000" 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:15.892440+0000", 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:16.336654+0000" 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.522 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:15.758009+0000", 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:16.336675+0000" 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:16.337178+0000", 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:16.748867+0000" 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:16.748868+0000", 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:16.749061+0000" 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.523 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:16.749062+0000", 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.426712+0000" 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:16.336676+0000", 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.426752+0000" 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:15.758006+0000", 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.426840+0000" 2024-04-25T08:04:14.524 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:15.757993+0000", 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.426848+0000" 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "60", 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.426849+0000", 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.427262+0000" 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.525 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "90", 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.427275+0000", 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.427299+0000" 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.427314+0000", 2024-04-25T08:04:14.526 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.528633+0000" 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.528635+0000", 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.528927+0000" 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.528928+0000", 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.528978+0000" 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.527 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.528978+0000", 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.979548+0000" 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.427307+0000", 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:17.979569+0000" 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.979855+0000", 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:18.248788+0000" 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.528 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:18.248789+0000", 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:18.248975+0000" 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:18.248976+0000", 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.940953+0000" 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.979570+0000", 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.940993+0000" 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.529 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.427300+0000", 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.941071+0000" 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:17.427262+0000", 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.941079+0000" 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "90", 2024-04-25T08:04:14.530 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.941080+0000", 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.941460+0000" 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "96", 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.941468+0000", 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.941492+0000" 2024-04-25T08:04:14.531 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.941507+0000", 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.988292+0000" 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.988294+0000", 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.988452+0000" 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.988453+0000", 2024-04-25T08:04:14.532 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.988484+0000" 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.988485+0000", 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:47.363259+0000" 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.941500+0000", 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:47.363275+0000" 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.533 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:47.363422+0000", 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:47.455117+0000" 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:47.455118+0000", 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:47.455173+0000" 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:47.455174+0000", 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.754822+0000" 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.534 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:47.363278+0000", 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.754885+0000" 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.941493+0000", 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.755174+0000" 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.941460+0000", 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.755190+0000" 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.535 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "97", 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:46.755192+0000", 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.756486+0000" 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "99", 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:46.756511+0000", 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.756576+0000" 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:46.756612+0000", 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.906389+0000" 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:46.906391+0000", 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.906563+0000" 2024-04-25T08:04:14.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:46.756596+0000", 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:46.906585+0000" 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:46.906586+0000", 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.379306+0000" 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:46.756577+0000", 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.379369+0000" 2024-04-25T08:04:14.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:46.756487+0000", 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.379379+0000" 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "99", 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.379380+0000", 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.379868+0000" 2024-04-25T08:04:14.539 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "113", 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.379881+0000", 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.379914+0000" 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.379934+0000", 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.381030+0000" 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.381031+0000", 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.381195+0000" 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.381196+0000", 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.381233+0000" 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.381234+0000", 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.713517+0000" 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.541 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.379924+0000", 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:47.713547+0000" 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.716548+0000", 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:48.301443+0000" 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:48.301445+0000", 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:48.301784+0000" 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.542 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:48.301785+0000", 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:48.302560+0000" 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:48.302563+0000", 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:22.717721+0000" 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:22.717722+0000", 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:22.717884+0000" 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.543 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:22.717885+0000", 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:34.495648+0000" 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:34.495649+0000", 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:34.496312+0000" 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:34.496317+0000", 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:00:19.303133+0000" 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.544 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:00:19.303134+0000", 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:00:19.846574+0000" 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.713548+0000", 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:00:19.846613+0000" 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.379914+0000", 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:00:19.846643+0000" 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.545 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:47.379869+0000", 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:00:19.846648+0000" 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "113", 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:00:19.846649+0000", 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:00:19.847052+0000" 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.546 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: "pg": "3.0", 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: "history": [ 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "40", 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.068377+0000", 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:06.068400+0000" 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.547 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.068413+0000", 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:06.202616+0000" 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.202618+0000", 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:06.202783+0000" 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.202783+0000", 2024-04-25T08:04:14.548 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:06.202806+0000" 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.202807+0000", 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:06.471976+0000" 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.068407+0000", 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:06.471987+0000" 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.472301+0000", 2024-04-25T08:04:14.549 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:06.742670+0000" 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.742672+0000", 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:06.742832+0000" 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.742832+0000", 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.300980+0000" 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.471988+0000", 2024-04-25T08:04:14.550 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.301017+0000" 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.068401+0000", 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.301094+0000" 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:06.068371+0000", 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.301100+0000" 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.551 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "40", 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.301101+0000", 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.301480+0000" 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "88", 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.301489+0000", 2024-04-25T08:04:14.552 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.301514+0000" 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.301529+0000", 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.411792+0000" 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.411794+0000", 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.411947+0000" 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.411948+0000", 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.411976+0000" 2024-04-25T08:04:14.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.411976+0000", 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.963619+0000" 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.301522+0000", 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:07.963642+0000" 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.963928+0000", 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:08.411073+0000" 2024-04-25T08:04:14.554 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:08.411075+0000", 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:08.411171+0000" 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:08.411172+0000", 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.485136+0000" 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.555 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.963643+0000", 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.485177+0000" 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.301515+0000", 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.485260+0000" 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:07.301481+0000", 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.485267+0000" 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.556 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "89", 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.485268+0000", 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.485774+0000" 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "105", 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.557 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.485783+0000", 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.485812+0000" 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.485832+0000", 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.644250+0000" 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.644252+0000", 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.644397+0000" 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.558 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.644398+0000", 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.644427+0000" 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.644428+0000", 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.942061+0000" 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.485822+0000", 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:46.942092+0000" 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.559 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.942258+0000", 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:47.120222+0000" 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:47.120223+0000", 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:47.120304+0000" 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:47.120304+0000", 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.724904+0000" 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.560 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.942093+0000", 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.724970+0000" 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.485813+0000", 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.725287+0000" 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:46.485775+0000", 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.725295+0000" 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.561 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "105", 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:07.725296+0000", 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.725948+0000" 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "107", 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.562 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:07.725961+0000", 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.725993+0000" 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:07.726011+0000", 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.727363+0000" 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:07.727364+0000", 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.727505+0000" 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.563 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:07.726002+0000", 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:07.727527+0000" 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:07.727527+0000", 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.133876+0000" 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:07.725994+0000", 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.133941+0000" 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.564 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:07.725949+0000", 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.133952+0000" 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "107", 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.133953+0000", 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.134460+0000" 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.565 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "117", 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.134473+0000", 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.134505+0000" 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.134524+0000", 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.248583+0000" 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.566 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.248584+0000", 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.248753+0000" 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.248754+0000", 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.248789+0000" 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.248790+0000", 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.513234+0000" 2024-04-25T08:04:14.567 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.134515+0000", 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:08.513256+0000" 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.517206+0000", 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:09.049063+0000" 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.568 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:09.049065+0000", 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:22.717841+0000" 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:22.717842+0000", 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:22.718401+0000" 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:22.718402+0000", 2024-04-25T08:04:14.569 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:34.495309+0000" 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:34.495310+0000", 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:34.495734+0000" 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:34.495735+0000", 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:02:04.280230+0000" 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:04:14.570 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:02:04.280230+0000", 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:02:04.280720+0000" 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:02:04.280720+0000", 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:04:07.919230+0000" 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:04:07.919231+0000", 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:04:08.144207+0000" 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.513257+0000", 2024-04-25T08:04:14.571 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:04:08.144260+0000" 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.134506+0000", 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:04:08.144314+0000" 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:08.134461+0000", 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:04:08.144325+0000" 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.572 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "117", 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:04:08.144326+0000", 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:04:08.144964+0000" 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: "pg": "1.0", 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: "history": [ 2024-04-25T08:04:14.573 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "27", 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Initial", 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.083884+0000", 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.084156+0000" 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "27", 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.574 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.084157+0000", 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.084251+0000" 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "30", 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.084259+0000", 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.084280+0000" 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.575 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.084300+0000", 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.168046+0000" 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.168048+0000", 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.168268+0000" 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.168270+0000", 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.168320+0000" 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.168321+0000", 2024-04-25T08:04:14.576 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.352007+0000" 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.084287+0000", 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.352022+0000" 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.352494+0000", 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.621836+0000" 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.621837+0000", 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:01.622002+0000" 2024-04-25T08:04:14.577 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.622002+0000", 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.341361+0000" 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.352025+0000", 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.341399+0000" 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.578 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.084281+0000", 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.341484+0000" 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:01.084252+0000", 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.341491+0000" 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.579 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "30", 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.341492+0000", 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.341912+0000" 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "101", 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.341920+0000", 2024-04-25T08:04:14.580 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.341944+0000" 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.341963+0000", 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.459710+0000" 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.459711+0000", 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.459877+0000" 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.581 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.459879+0000", 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.459910+0000" 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.459911+0000", 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.659988+0000" 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.341953+0000", 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:02.660007+0000" 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.582 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.660300+0000", 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:03.380837+0000" 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:03.380838+0000", 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:52:03.380932+0000" 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:03.380932+0000", 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.442189+0000" 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.583 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.660007+0000", 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.442209+0000" 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.341945+0000", 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.442342+0000" 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:52:02.341912+0000", 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.442345+0000" 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.584 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "101", 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:59.442346+0000", 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.442599+0000" 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "103", 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.585 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:59.442602+0000", 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.442613+0000" 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:59.442619+0000", 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.569237+0000" 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:59.569238+0000", 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.569401+0000" 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.586 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:59.442616+0000", 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:57:59.569420+0000" 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:59.569421+0000", 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.080135+0000" 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:59.442613+0000", 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.080192+0000" 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.587 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:57:59.442599+0000", 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.080202+0000" 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "103", 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.080203+0000", 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.080687+0000" 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.588 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "115", 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Start", 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.080701+0000", 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.080730+0000" 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.080750+0000", 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.193778+0000" 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:04:14.589 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.193780+0000", 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.193977+0000" 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.193978+0000", 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.194014+0000" 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.194015+0000", 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.532819+0000" 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.590 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.080740+0000", 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.532851+0000" 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.535959+0000", 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T07:58:00.977768+0000" 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.977770+0000", 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:00:19.303257+0000" 2024-04-25T08:04:14.591 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:00:19.303257+0000", 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:00:19.303707+0000" 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:00:19.303708+0000", 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:02:04.280010+0000" 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:02:04.280011+0000", 2024-04-25T08:04:14.592 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:02:04.948706+0000" 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary/Active", 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.532852+0000", 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:02:04.948759+0000" 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started/Primary", 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.080731+0000", 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:02:04.948807+0000" 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Started", 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T07:58:00.080688+0000", 2024-04-25T08:04:14.593 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:02:04.948816+0000" 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": "115", 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: "states": [ 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "Reset", 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: "enter": "2024-04-25T08:02:04.948817+0000", 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: "exit": "2024-04-25T08:02:04.949473+0000" 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.594 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:04:14.595 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:04:14.595 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:04:14.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:302: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2024-04-25T08:04:14.595 INFO:tasks.workunit.client.0.smithi088.stdout:TEST PASSED 2024-04-25T08:04:14.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: echo TEST PASSED 2024-04-25T08:04:14.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: delete_pool test1 2024-04-25T08:04:14.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:04:14.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:04:15.099 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test1' does not exist 2024-04-25T08:04:15.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: delete_pool test3 2024-04-25T08:04:15.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-04-25T08:04:15.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-04-25T08:04:15.735 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test3' does not exist 2024-04-25T08:04:15.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:311: TEST_recovery_priority: delete_pool test7 2024-04-25T08:04:15.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2024-04-25T08:04:15.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2024-04-25T08:04:16.640 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test7' does not exist 2024-04-25T08:04:16.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:312: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2024-04-25T08:04:16.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:04:16.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:04:16.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:04:16.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:04:16.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:04:21.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:04:21.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:313: TEST_recovery_priority: return 0 2024-04-25T08:04:21.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-04-25T08:04:21.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:04:21.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:04:21.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:04:21.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:04:21.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:04:21.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:04:21.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:04:21.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:04:21.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:04:21.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:04:21.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:04:21.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:04:21.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:04:21.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:04:21.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:04:21.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:04:21.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:04:21.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:04:21.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:04:21.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:04:21.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:04:21.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:04:22.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:04:22.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:22.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T08:04:22.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202479 2024-04-25T08:04:22.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:04:22.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:04:22.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:04:22.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-prio 0 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:04:22.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:04:22.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:04:22.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:04:22.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:04:22.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:04:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:04:22.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:04:22.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:04:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:04:22.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:04:22.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:04:22.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:04:22.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:04:22.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:04:22.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:04:22.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:22.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202479 2024-04-25T08:04:22.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202479 2024-04-25T08:04:22.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:04:22.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:04:22.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:04:22.088 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:04:22.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:04:22.098 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2024-04-25T08:04:22.098 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2024-04-25T08:04:22.151 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:04:22.157 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:04:22.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:04:22.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:04:22.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:04:22.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-space 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=25760e63-e148-4ff9-90a2-51e52013dc4d --auth-supported=none ' 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2024-04-25T08:04:22.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2024-04-25T08:04:22.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-04-25T08:04:22.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export objects=600 2024-04-25T08:04:22.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: objects=600 2024-04-25T08:04:22.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: export poolprefix=test 2024-04-25T08:04:22.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: poolprefix=test 2024-04-25T08:04:22.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: set 2024-04-25T08:04:22.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:04:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: local funcs=TEST_recovery_test_simple 2024-04-25T08:04:22.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: for func in $funcs 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: setup td/osd-recovery-space 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:04:22.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:04:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:04:22.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:04:22.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:04:22.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:04:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:04:22.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:04:22.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:04:22.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:04:22.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:04:22.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:04:22.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:04:22.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:04:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:04:22.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:04:22.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:04:22.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:22.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:04:22.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258125 2024-04-25T08:04:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:04:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:04:22.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2024-04-25T08:04:22.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:04:22.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:22.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:04:22.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.258125 2024-04-25T08:04:22.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: TEST_recovery_test_simple td/osd-recovery-space 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local pools=1 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:91: TEST_recovery_test_simple: local OSDS=2 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2024-04-25T08:04:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2024-04-25T08:04:22.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:04:22.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:04:22.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:04:22.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2024-04-25T08:04:22.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2024-04-25T08:04:22.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:04:22.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:04:22.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:04:22.724 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:04:22.724 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:22.724 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:04:22.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258125/$cluster-$name.asok' 2024-04-25T08:04:22.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.258125/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:04:22.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:04:22.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:04:22.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:04:22.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:04:22.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:04:22.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:04:22.962 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:04:22.962 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:04:22.962 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:04:22.963 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:04:22.963 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:22.963 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:04:22.963 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.258125/ceph-mon.a.asok 2024-04-25T08:04:22.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:04:22.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.258125/ceph-mon.a.asok config get fsid 2024-04-25T08:04:23.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:04:23.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:04:23.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:04:23.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:04:23.080 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.258125/ceph-mon.a.asok 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:04:23.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.258125/ceph-mon.a.asok config get mon_host 2024-04-25T08:04:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2024-04-25T08:04:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2024-04-25T08:04:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:04:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:04:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:04:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2024-04-25T08:04:23.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:04:24.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:04:24.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:04:24.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:04:24.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:04:24.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:24.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:04:24.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258125/$cluster-$name.asok' 2024-04-25T08:04:24.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:04:24.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.258125/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:04:24.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:95: TEST_recovery_test_simple: export CEPH_ARGS 2024-04-25T08:04:24.184 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: expr 2 - 1 2024-04-25T08:04:24.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: seq 0 1 2024-04-25T08:04:24.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:04:24.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2024-04-25T08:04:24.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-04-25T08:04:24.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:04:24.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:04:24.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:04:24.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=25760e63-e148-4ff9-90a2-51e52013dc4d --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:04:24.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258125/$cluster-$name.asok' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.258125/$cluster-$name.asok' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:04:24.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:04:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:04:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:04:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:04:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2024-04-25T08:04:24.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:04:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3af4a615-6f05-40c2-8d5a-84e39f2f0e01 2024-04-25T08:04:24.192 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 3af4a615-6f05-40c2-8d5a-84e39f2f0e01 2024-04-25T08:04:24.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3af4a615-6f05-40c2-8d5a-84e39f2f0e01' 2024-04-25T08:04:24.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:04:24.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIDipmB8ejDBAAKvYyFaAO2mLwaa6mAsl7Xw== 2024-04-25T08:04:24.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIDipmB8ejDBAAKvYyFaAO2mLwaa6mAsl7Xw=="}' 2024-04-25T08:04:24.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3af4a615-6f05-40c2-8d5a-84e39f2f0e01 -i td/osd-recovery-space/0/new.json 2024-04-25T08:04:24.562 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:04:24.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2024-04-25T08:04:24.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=25760e63-e148-4ff9-90a2-51e52013dc4d --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.258125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIDipmB8ejDBAAKvYyFaAO2mLwaa6mAsl7Xw== --osd-uuid 3af4a615-6f05-40c2-8d5a-84e39f2f0e01 2024-04-25T08:04:24.615 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:24.616+0000 7f2596f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:24.647 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:24.648+0000 7f2596f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:24.649 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:24.648+0000 7f2596f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:24.649 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:24.648+0000 7f2596f397c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:04:24.649 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:24.648+0000 7f2596f397c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:04:24.649 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:24.648+0000 7f2596f397c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:04:24.687 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:24.688+0000 7f2596f397c0 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2024-04-25T08:04:32.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2024-04-25T08:04:32.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:04:32.696 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:04:32.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:04:32.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:04:33.305 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:04:33.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:04:33.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=25760e63-e148-4ff9-90a2-51e52013dc4d --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.258125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:04:33.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:04:33.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:04:33.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:04:33.338 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:33.340+0000 7fd1354ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:33.368 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:33.368+0000 7fd1354ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:33.370 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:33.372+0000 7fd1354ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:33.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:04:33.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:04:33.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:04:33.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:04:33.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:04:33.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:33.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:04:33.693 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:04:33.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:33.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:33.924 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:33.924+0000 7fd1354ce7c0 -1 Falling back to public interface 2024-04-25T08:04:34.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:35.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:35.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:35.151 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:04:35.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:04:35.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:35.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:35.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:36.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:36.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:36.554 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:04:36.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:04:36.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:36.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:36.741 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:04:36.740+0000 7fd1354ce7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:04:37.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:38.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:04:38.029 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:04:38.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:38.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:38.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:39.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:39.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:39.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:04:39.480 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:04:39.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:39.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:39.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:40.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:40.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:40.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:04:40.977 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:04:40.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:40.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:41.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:42.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:42.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:42.431 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:04:42.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:04:42.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:42.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:43.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:44.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:44.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:44.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:04:44.184 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:04:44.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:44.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:44.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:45.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:45.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:45.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:04:45.867 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:04:45.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:45.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:46.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:47.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:47.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:47.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:04:47.309 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:04:47.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:47.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:47.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:48.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:48.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:48.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:04:48.916 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:04:48.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:48.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:49.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:50.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:50.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:50.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:04:50.540 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:04:50.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:50.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:50.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:51.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:51.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:51.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:04:51.964 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:04:51.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:51.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:52.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:53.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:53.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:53.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:04:53.409 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:04:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:54.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:55.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:55.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:55.047 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:04:55.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:04:55.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:55.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:55.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:56.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:56.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:56.671 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:04:56.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:04:56.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:56.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:57.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:58.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:58.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:58.688 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:04:58.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:04:58.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:58.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:59.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:00.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:00.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:00.791 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:05:00.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:05:00.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:00.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:05:02.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:03.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:03.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:03.149 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:05:03.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:05:03.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:03.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:05:03.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:04.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:04.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:04.908 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:05:04.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:05:04.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:04.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:05:05.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:06.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:06.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:06.441 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:05:06.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:05:06.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:06.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:05:07.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:08.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:08.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:08.337 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:05:08.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:05:08.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:08.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:05:08.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:09.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:09.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:09.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:05:09.931 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:05:09.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:09.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:05:10.341 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/768370771,v1:127.0.0.1:6803/768370771] [v2:127.0.0.1:6804/768370771,v1:127.0.0.1:6805/768370771] exists,up 3af4a615-6f05-40c2-8d5a-84e39f2f0e01 2024-04-25T08:05:10.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=25760e63-e148-4ff9-90a2-51e52013dc4d --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:05:10.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.258125/$cluster-$name.asok' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.258125/$cluster-$name.asok' 2024-04-25T08:05:10.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2024-04-25T08:05:10.348 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:05:10.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=91989f59-6b1e-44f0-9d3e-1ba12dbe368b 2024-04-25T08:05:10.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 91989f59-6b1e-44f0-9d3e-1ba12dbe368b' 2024-04-25T08:05:10.349 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 91989f59-6b1e-44f0-9d3e-1ba12dbe368b 2024-04-25T08:05:10.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:05:10.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2DipmvmL1FRAACjVbjUG5TyOoXBWOJT6SRA== 2024-04-25T08:05:10.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2DipmvmL1FRAACjVbjUG5TyOoXBWOJT6SRA=="}' 2024-04-25T08:05:10.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 91989f59-6b1e-44f0-9d3e-1ba12dbe368b -i td/osd-recovery-space/1/new.json 2024-04-25T08:05:10.921 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:05:10.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2024-04-25T08:05:10.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=25760e63-e148-4ff9-90a2-51e52013dc4d --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.258125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2DipmvmL1FRAACjVbjUG5TyOoXBWOJT6SRA== --osd-uuid 91989f59-6b1e-44f0-9d3e-1ba12dbe368b 2024-04-25T08:05:10.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:10.976+0000 7f2b49c6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:10.977 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:10.976+0000 7f2b49c6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:10.978 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:10.980+0000 7f2b49c6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:10.980+0000 7f2b49c6e7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:05:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:10.980+0000 7f2b49c6e7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:05:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:10.980+0000 7f2b49c6e7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:05:11.081 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:11.084+0000 7f2b49c6e7c0 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2024-04-25T08:05:19.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2024-04-25T08:05:19.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:05:19.196 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:05:19.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:05:19.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:05:19.761 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:05:19.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:05:19.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=25760e63-e148-4ff9-90a2-51e52013dc4d --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.258125/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:05:19.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:05:19.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:05:19.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:05:19.794 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:19.795+0000 7f6a424ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:19.797 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:19.795+0000 7f6a424ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:19.798 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:19.799+0000 7f6a424ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:20.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:05:20.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:05:20.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:05:20.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:05:20.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:05:20.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:20.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:05:20.177 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:05:20.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:20.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:20.363 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:20.363+0000 7f6a424ed7c0 -1 Falling back to public interface 2024-04-25T08:05:20.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:21.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:21.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:21.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:05:21.624 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:05:21.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:21.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:22.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:23.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:23.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:23.030 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:05:23.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:05:23.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:23.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:23.278 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:05:23.279+0000 7f6a424ed7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:05:23.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:24.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:24.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:24.562 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:05:24.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:05:24.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:24.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:25.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:26.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:26.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:26.402 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:05:26.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:05:26.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:26.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:26.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:27.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:27.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:27.989 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:05:27.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:05:27.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:27.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:28.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:29.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:29.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:29.545 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:05:29.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:05:29.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:29.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:30.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:31.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:31.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:31.191 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:05:31.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:05:31.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:31.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:31.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:32.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:32.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:32.738 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:05:32.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:05:32.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:32.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:33.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:34.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:34.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:34.205 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:05:34.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:05:34.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:34.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:34.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:35.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:35.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:35.850 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:05:35.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:05:35.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:35.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:36.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:37.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:37.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:37.357 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:05:37.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:05:37.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:37.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:38.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:39.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:39.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:39.061 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:05:39.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:05:39.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:39.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:39.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:40.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:40.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:40.482 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:05:40.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:05:40.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:40.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:40.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:41.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:41.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:41.980 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:05:41.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:05:41.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:41.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:42.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:43.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:43.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:43.589 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:05:43.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:05:43.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:43.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:44.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:45.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:45.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:45.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:05:45.702 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:05:45.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:45.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:46.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:47.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:47.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:47.846 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:05:47.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:05:47.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:47.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:48.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:49.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:49.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:49.527 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:05:49.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:05:49.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:49.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:50.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:51.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:51.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:51.647 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:05:51.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:05:51.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:51.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:52.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:53.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:53.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:53.857 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:05:53.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:05:53.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:53.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:54.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:55.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:55.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:55.622 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:05:55.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:05:55.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:55.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:56.059 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1778629063,v1:127.0.0.1:6811/1778629063] [v2:127.0.0.1:6812/1778629063,v1:127.0.0.1:6813/1778629063] exists,up 91989f59-6b1e-44f0-9d3e-1ba12dbe368b 2024-04-25T08:05:56.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:05:56.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:05:56.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:05:56.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2024-04-25T08:05:57.139 INFO:tasks.workunit.client.0.smithi088.stderr:osd set-nearfull-ratio 0.4 2024-04-25T08:05:57.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2024-04-25T08:05:58.242 INFO:tasks.workunit.client.0.smithi088.stderr:osd set-backfillfull-ratio 0.45 2024-04-25T08:05:58.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:104: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2024-04-25T08:05:59.220 INFO:tasks.workunit.client.0.smithi088.stderr:osd set-full-ratio 0.5 2024-04-25T08:05:59.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:05:59.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-04-25T08:05:59.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: create_pool test1 1 1 2024-04-25T08:05:59.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:06:00.096 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test1' already exists 2024-04-25T08:06:00.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:06:01.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:109: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-04-25T08:06:02.172 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 1 2024-04-25T08:06:02.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:112: TEST_recovery_test_simple: wait_for_clean 2024-04-25T08:06:02.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:06:02.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:06:02.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:06:02.196 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:06:02.197 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:06:02.197 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:06:02.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:06:02.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:06:02.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:06:02.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:06:02.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:06:02.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:06:02.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:06:02.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:06:02.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:06:02.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:06:02.895 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T08:06:02.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:06:02.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:06:02.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:06:03.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836499 2024-04-25T08:06:03.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836499 2024-04-25T08:06:03.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499' 2024-04-25T08:06:03.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:06:03.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:06:03.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672973 2024-04-25T08:06:03.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672973 2024-04-25T08:06:03.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672973' 2024-04-25T08:06:03.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:06:03.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:06:03.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836499 2024-04-25T08:06:03.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:06:03.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836499 2024-04-25T08:06:03.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:06:03.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836499 2024-04-25T08:06:03.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-04-25T08:06:03.301 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836499 2024-04-25T08:06:03.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:06:03.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836497 -lt 21474836499 2024-04-25T08:06:03.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:06:04.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:06:04.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:06:05.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836499 2024-04-25T08:06:05.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:06:05.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672973 2024-04-25T08:06:05.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:06:05.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:06:05.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672973 2024-04-25T08:06:05.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:06:05.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672973 2024-04-25T08:06:05.189 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672973 2024-04-25T08:06:05.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-04-25T08:06:05.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:06:05.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672973 2024-04-25T08:06:05.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:06:05.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:06:05.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:06:06.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:06:06.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:06:06.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:06:06.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:06:06.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:06:06.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:06:06.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:06:06.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:06:06.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:06:06.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:06:06.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:06:06.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:06:07.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:06:07.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:06:07.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:06:07.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2024-04-25T08:06:07.235 INFO:tasks.workunit.client.0.smithi088.stderr:5+0 records in 2024-04-25T08:06:07.235 INFO:tasks.workunit.client.0.smithi088.stderr:5+0 records out 2024-04-25T08:06:07.235 INFO:tasks.workunit.client.0.smithi088.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 0.000138156 s, 37.1 MB/s 2024-04-25T08:06:07.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: seq 1 600 2024-04-25T08:06:07.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:07.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2024-04-25T08:06:07.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:07.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2024-04-25T08:06:07.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:07.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2024-04-25T08:06:07.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:07.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2024-04-25T08:06:07.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:07.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2024-04-25T08:06:07.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:07.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2024-04-25T08:06:08.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:08.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2024-04-25T08:06:08.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:08.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2024-04-25T08:06:08.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:08.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2024-04-25T08:06:08.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:08.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2024-04-25T08:06:08.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:08.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2024-04-25T08:06:08.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:08.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2024-04-25T08:06:08.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:08.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2024-04-25T08:06:08.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:08.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2024-04-25T08:06:09.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:09.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2024-04-25T08:06:09.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:09.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2024-04-25T08:06:09.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:09.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2024-04-25T08:06:09.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:09.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2024-04-25T08:06:09.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:09.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2024-04-25T08:06:09.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:09.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2024-04-25T08:06:09.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:09.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2024-04-25T08:06:10.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:10.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2024-04-25T08:06:10.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:10.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2024-04-25T08:06:10.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:10.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2024-04-25T08:06:10.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:10.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2024-04-25T08:06:10.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:10.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2024-04-25T08:06:10.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:10.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2024-04-25T08:06:10.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:10.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2024-04-25T08:06:10.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:10.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2024-04-25T08:06:11.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:11.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2024-04-25T08:06:11.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:11.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2024-04-25T08:06:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:11.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2024-04-25T08:06:11.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:11.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2024-04-25T08:06:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:11.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2024-04-25T08:06:11.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:11.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2024-04-25T08:06:11.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:11.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2024-04-25T08:06:12.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:12.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2024-04-25T08:06:12.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:12.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2024-04-25T08:06:12.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:12.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2024-04-25T08:06:12.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2024-04-25T08:06:12.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:12.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2024-04-25T08:06:12.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:12.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2024-04-25T08:06:12.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:12.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2024-04-25T08:06:12.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:12.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2024-04-25T08:06:13.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:13.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2024-04-25T08:06:13.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:13.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2024-04-25T08:06:13.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:13.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2024-04-25T08:06:13.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:13.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2024-04-25T08:06:13.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:13.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2024-04-25T08:06:13.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:13.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2024-04-25T08:06:13.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:13.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2024-04-25T08:06:14.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:14.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2024-04-25T08:06:14.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:14.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2024-04-25T08:06:14.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:14.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2024-04-25T08:06:14.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:14.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2024-04-25T08:06:14.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:14.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2024-04-25T08:06:14.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:14.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2024-04-25T08:06:14.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:14.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2024-04-25T08:06:14.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:14.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2024-04-25T08:06:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2024-04-25T08:06:15.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:15.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2024-04-25T08:06:15.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:15.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2024-04-25T08:06:15.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:15.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2024-04-25T08:06:15.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:15.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2024-04-25T08:06:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:15.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2024-04-25T08:06:15.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:15.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2024-04-25T08:06:16.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:16.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2024-04-25T08:06:16.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:16.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2024-04-25T08:06:16.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:16.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2024-04-25T08:06:16.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:16.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2024-04-25T08:06:16.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:16.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2024-04-25T08:06:16.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:16.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2024-04-25T08:06:16.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:16.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2024-04-25T08:06:16.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:16.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2024-04-25T08:06:17.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:17.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2024-04-25T08:06:17.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:17.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2024-04-25T08:06:17.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:17.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2024-04-25T08:06:17.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:17.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2024-04-25T08:06:17.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:17.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2024-04-25T08:06:17.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:17.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2024-04-25T08:06:17.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:17.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2024-04-25T08:06:18.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:18.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2024-04-25T08:06:18.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:18.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2024-04-25T08:06:18.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:18.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2024-04-25T08:06:18.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:18.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2024-04-25T08:06:18.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:18.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2024-04-25T08:06:18.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:18.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2024-04-25T08:06:18.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:18.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2024-04-25T08:06:18.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:18.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2024-04-25T08:06:19.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:19.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2024-04-25T08:06:19.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:19.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2024-04-25T08:06:19.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:19.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2024-04-25T08:06:19.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:19.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2024-04-25T08:06:19.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:19.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2024-04-25T08:06:19.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:19.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2024-04-25T08:06:19.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:19.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2024-04-25T08:06:20.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:20.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2024-04-25T08:06:20.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:20.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2024-04-25T08:06:20.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:20.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2024-04-25T08:06:20.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:20.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2024-04-25T08:06:20.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:20.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2024-04-25T08:06:20.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:20.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2024-04-25T08:06:21.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:21.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2024-04-25T08:06:21.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:21.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2024-04-25T08:06:21.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:21.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2024-04-25T08:06:21.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:21.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2024-04-25T08:06:21.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:21.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2024-04-25T08:06:21.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:21.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2024-04-25T08:06:21.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:21.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2024-04-25T08:06:22.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:22.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2024-04-25T08:06:22.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:22.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2024-04-25T08:06:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:22.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2024-04-25T08:06:22.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:22.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2024-04-25T08:06:22.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:22.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2024-04-25T08:06:22.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:22.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2024-04-25T08:06:22.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2024-04-25T08:06:22.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:22.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2024-04-25T08:06:23.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:23.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2024-04-25T08:06:23.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:23.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2024-04-25T08:06:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2024-04-25T08:06:23.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:23.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2024-04-25T08:06:23.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:23.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2024-04-25T08:06:23.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:23.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2024-04-25T08:06:23.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:23.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2024-04-25T08:06:24.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:24.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2024-04-25T08:06:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2024-04-25T08:06:24.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:24.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2024-04-25T08:06:24.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:24.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2024-04-25T08:06:24.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:24.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2024-04-25T08:06:24.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:24.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2024-04-25T08:06:24.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:24.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2024-04-25T08:06:25.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:25.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2024-04-25T08:06:25.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:25.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2024-04-25T08:06:25.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:25.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2024-04-25T08:06:25.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:25.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2024-04-25T08:06:25.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:25.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2024-04-25T08:06:25.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:25.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2024-04-25T08:06:26.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:26.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2024-04-25T08:06:26.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:26.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2024-04-25T08:06:26.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:26.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2024-04-25T08:06:26.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:26.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2024-04-25T08:06:26.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:26.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2024-04-25T08:06:26.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:26.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2024-04-25T08:06:26.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:26.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2024-04-25T08:06:26.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:26.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2024-04-25T08:06:27.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:27.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2024-04-25T08:06:27.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:27.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2024-04-25T08:06:27.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:27.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2024-04-25T08:06:27.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:27.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2024-04-25T08:06:27.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:27.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2024-04-25T08:06:27.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:27.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2024-04-25T08:06:27.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:27.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2024-04-25T08:06:28.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:28.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2024-04-25T08:06:28.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:28.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2024-04-25T08:06:28.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:28.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2024-04-25T08:06:28.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:28.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2024-04-25T08:06:28.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:28.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2024-04-25T08:06:28.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:28.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2024-04-25T08:06:28.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:28.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2024-04-25T08:06:28.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:28.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2024-04-25T08:06:29.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:29.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2024-04-25T08:06:29.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:29.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2024-04-25T08:06:29.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:29.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2024-04-25T08:06:29.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:29.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2024-04-25T08:06:29.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:29.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2024-04-25T08:06:29.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:29.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2024-04-25T08:06:29.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:29.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2024-04-25T08:06:30.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:30.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2024-04-25T08:06:30.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:30.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2024-04-25T08:06:30.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:30.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2024-04-25T08:06:30.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:30.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2024-04-25T08:06:30.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:30.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2024-04-25T08:06:30.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:30.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2024-04-25T08:06:30.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:30.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2024-04-25T08:06:30.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:30.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2024-04-25T08:06:31.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:31.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2024-04-25T08:06:31.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:31.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2024-04-25T08:06:31.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:31.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2024-04-25T08:06:31.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:31.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2024-04-25T08:06:31.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:31.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2024-04-25T08:06:31.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:31.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2024-04-25T08:06:31.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:31.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2024-04-25T08:06:32.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:32.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2024-04-25T08:06:32.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:32.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2024-04-25T08:06:32.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:32.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2024-04-25T08:06:32.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:32.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2024-04-25T08:06:32.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:32.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2024-04-25T08:06:32.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:32.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2024-04-25T08:06:32.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:32.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2024-04-25T08:06:32.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:32.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2024-04-25T08:06:33.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:33.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2024-04-25T08:06:33.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:33.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2024-04-25T08:06:33.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:33.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2024-04-25T08:06:33.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:33.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2024-04-25T08:06:33.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:33.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2024-04-25T08:06:33.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:33.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2024-04-25T08:06:33.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:33.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2024-04-25T08:06:34.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:34.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2024-04-25T08:06:34.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:34.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2024-04-25T08:06:34.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:34.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2024-04-25T08:06:34.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:34.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2024-04-25T08:06:34.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:34.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2024-04-25T08:06:34.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:34.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2024-04-25T08:06:34.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:34.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2024-04-25T08:06:35.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:35.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2024-04-25T08:06:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2024-04-25T08:06:35.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:35.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2024-04-25T08:06:35.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:35.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2024-04-25T08:06:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:35.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2024-04-25T08:06:35.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:35.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2024-04-25T08:06:35.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:35.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2024-04-25T08:06:35.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:35.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2024-04-25T08:06:36.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:36.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2024-04-25T08:06:36.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:36.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2024-04-25T08:06:36.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:36.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2024-04-25T08:06:36.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:36.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2024-04-25T08:06:36.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:36.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2024-04-25T08:06:36.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:36.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2024-04-25T08:06:36.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:36.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2024-04-25T08:06:36.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:36.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2024-04-25T08:06:37.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:37.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2024-04-25T08:06:37.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:37.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2024-04-25T08:06:37.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:37.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2024-04-25T08:06:37.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:37.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2024-04-25T08:06:37.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:37.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2024-04-25T08:06:37.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:37.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2024-04-25T08:06:37.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:37.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2024-04-25T08:06:38.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:38.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2024-04-25T08:06:38.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:38.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2024-04-25T08:06:38.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:38.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2024-04-25T08:06:38.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:38.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2024-04-25T08:06:38.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:38.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2024-04-25T08:06:38.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:38.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2024-04-25T08:06:38.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:38.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2024-04-25T08:06:38.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:38.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2024-04-25T08:06:39.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:39.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2024-04-25T08:06:39.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:39.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2024-04-25T08:06:39.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:39.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2024-04-25T08:06:39.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:39.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2024-04-25T08:06:39.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:39.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2024-04-25T08:06:39.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:39.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2024-04-25T08:06:39.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:39.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2024-04-25T08:06:40.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:40.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2024-04-25T08:06:40.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:40.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2024-04-25T08:06:40.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:40.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2024-04-25T08:06:40.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:40.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2024-04-25T08:06:40.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:40.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2024-04-25T08:06:40.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:40.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2024-04-25T08:06:40.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:40.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2024-04-25T08:06:40.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:40.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2024-04-25T08:06:41.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:41.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2024-04-25T08:06:41.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:41.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2024-04-25T08:06:41.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:41.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2024-04-25T08:06:41.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:41.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2024-04-25T08:06:41.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:41.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2024-04-25T08:06:41.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:41.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2024-04-25T08:06:41.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:41.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2024-04-25T08:06:42.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:42.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2024-04-25T08:06:42.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:42.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2024-04-25T08:06:42.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:42.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2024-04-25T08:06:42.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:42.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2024-04-25T08:06:42.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:42.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2024-04-25T08:06:42.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:42.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2024-04-25T08:06:42.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:42.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2024-04-25T08:06:43.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:43.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2024-04-25T08:06:43.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:43.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2024-04-25T08:06:43.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:43.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2024-04-25T08:06:43.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:43.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2024-04-25T08:06:43.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:43.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2024-04-25T08:06:43.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:43.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2024-04-25T08:06:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2024-04-25T08:06:43.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:43.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2024-04-25T08:06:44.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:44.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2024-04-25T08:06:44.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:44.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2024-04-25T08:06:44.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:44.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2024-04-25T08:06:44.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:44.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2024-04-25T08:06:44.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:44.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2024-04-25T08:06:44.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:44.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2024-04-25T08:06:44.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:44.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2024-04-25T08:06:44.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:44.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2024-04-25T08:06:45.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:45.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2024-04-25T08:06:45.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:45.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2024-04-25T08:06:45.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:45.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2024-04-25T08:06:45.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:45.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2024-04-25T08:06:45.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:45.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2024-04-25T08:06:45.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:45.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2024-04-25T08:06:45.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:45.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2024-04-25T08:06:46.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:46.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2024-04-25T08:06:46.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:46.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2024-04-25T08:06:46.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:46.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2024-04-25T08:06:46.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:46.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2024-04-25T08:06:46.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:46.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2024-04-25T08:06:46.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:46.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2024-04-25T08:06:46.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:46.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2024-04-25T08:06:46.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:46.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2024-04-25T08:06:47.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:47.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2024-04-25T08:06:47.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:47.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2024-04-25T08:06:47.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:47.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2024-04-25T08:06:47.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:47.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2024-04-25T08:06:47.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:47.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2024-04-25T08:06:47.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:47.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2024-04-25T08:06:47.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:47.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2024-04-25T08:06:48.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:48.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2024-04-25T08:06:48.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:48.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2024-04-25T08:06:48.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:48.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2024-04-25T08:06:48.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:48.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2024-04-25T08:06:48.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:48.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2024-04-25T08:06:48.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:48.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2024-04-25T08:06:48.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:48.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2024-04-25T08:06:48.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:48.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2024-04-25T08:06:49.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:49.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2024-04-25T08:06:49.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:49.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2024-04-25T08:06:49.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:49.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2024-04-25T08:06:49.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:49.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2024-04-25T08:06:49.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:49.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2024-04-25T08:06:49.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:49.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2024-04-25T08:06:49.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:49.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2024-04-25T08:06:50.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:50.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2024-04-25T08:06:50.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:50.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2024-04-25T08:06:50.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:50.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2024-04-25T08:06:50.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:50.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2024-04-25T08:06:50.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:50.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2024-04-25T08:06:50.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:50.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2024-04-25T08:06:50.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:50.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2024-04-25T08:06:51.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2024-04-25T08:06:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2024-04-25T08:06:51.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2024-04-25T08:06:51.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2024-04-25T08:06:51.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2024-04-25T08:06:51.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2024-04-25T08:06:51.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2024-04-25T08:06:51.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2024-04-25T08:06:51.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:51.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2024-04-25T08:06:52.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2024-04-25T08:06:52.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2024-04-25T08:06:52.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2024-04-25T08:06:52.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2024-04-25T08:06:52.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2024-04-25T08:06:52.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2024-04-25T08:06:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2024-04-25T08:06:52.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2024-04-25T08:06:52.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2024-04-25T08:06:52.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:52.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2024-04-25T08:06:53.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2024-04-25T08:06:53.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2024-04-25T08:06:53.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2024-04-25T08:06:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2024-04-25T08:06:53.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2024-04-25T08:06:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2024-04-25T08:06:53.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2024-04-25T08:06:53.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2024-04-25T08:06:53.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2024-04-25T08:06:53.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:53.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2024-04-25T08:06:54.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2024-04-25T08:06:54.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2024-04-25T08:06:54.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2024-04-25T08:06:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2024-04-25T08:06:54.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2024-04-25T08:06:54.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2024-04-25T08:06:54.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2024-04-25T08:06:54.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2024-04-25T08:06:54.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:54.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2024-04-25T08:06:55.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2024-04-25T08:06:55.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2024-04-25T08:06:55.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2024-04-25T08:06:55.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2024-04-25T08:06:55.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2024-04-25T08:06:55.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2024-04-25T08:06:55.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2024-04-25T08:06:55.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2024-04-25T08:06:55.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2024-04-25T08:06:55.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:55.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2024-04-25T08:06:56.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:56.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2024-04-25T08:06:56.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:56.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2024-04-25T08:06:56.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:56.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2024-04-25T08:06:56.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:56.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2024-04-25T08:06:56.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:56.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2024-04-25T08:06:56.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:56.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2024-04-25T08:06:56.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:56.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2024-04-25T08:06:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2024-04-25T08:06:57.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2024-04-25T08:06:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2024-04-25T08:06:57.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2024-04-25T08:06:57.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2024-04-25T08:06:57.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2024-04-25T08:06:57.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2024-04-25T08:06:57.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2024-04-25T08:06:57.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:57.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2024-04-25T08:06:58.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2024-04-25T08:06:58.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2024-04-25T08:06:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2024-04-25T08:06:58.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2024-04-25T08:06:58.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2024-04-25T08:06:58.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2024-04-25T08:06:58.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2024-04-25T08:06:58.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2024-04-25T08:06:58.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2024-04-25T08:06:58.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2024-04-25T08:06:58.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:58.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2024-04-25T08:06:59.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2024-04-25T08:06:59.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2024-04-25T08:06:59.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2024-04-25T08:06:59.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2024-04-25T08:06:59.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2024-04-25T08:06:59.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2024-04-25T08:06:59.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2024-04-25T08:06:59.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2024-04-25T08:06:59.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2024-04-25T08:06:59.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:06:59.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2024-04-25T08:07:00.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2024-04-25T08:07:00.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2024-04-25T08:07:00.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2024-04-25T08:07:00.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2024-04-25T08:07:00.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2024-04-25T08:07:00.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2024-04-25T08:07:00.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2024-04-25T08:07:00.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2024-04-25T08:07:00.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:00.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2024-04-25T08:07:01.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2024-04-25T08:07:01.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2024-04-25T08:07:01.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2024-04-25T08:07:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2024-04-25T08:07:01.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2024-04-25T08:07:01.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2024-04-25T08:07:01.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2024-04-25T08:07:01.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2024-04-25T08:07:01.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2024-04-25T08:07:01.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:01.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2024-04-25T08:07:02.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2024-04-25T08:07:02.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2024-04-25T08:07:02.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2024-04-25T08:07:02.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2024-04-25T08:07:02.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2024-04-25T08:07:02.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2024-04-25T08:07:02.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2024-04-25T08:07:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2024-04-25T08:07:02.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2024-04-25T08:07:02.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:02.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2024-04-25T08:07:03.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2024-04-25T08:07:03.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2024-04-25T08:07:03.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2024-04-25T08:07:03.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2024-04-25T08:07:03.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2024-04-25T08:07:03.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2024-04-25T08:07:03.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2024-04-25T08:07:03.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2024-04-25T08:07:03.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2024-04-25T08:07:03.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:03.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2024-04-25T08:07:04.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2024-04-25T08:07:04.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2024-04-25T08:07:04.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2024-04-25T08:07:04.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2024-04-25T08:07:04.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2024-04-25T08:07:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2024-04-25T08:07:04.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2024-04-25T08:07:04.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2024-04-25T08:07:04.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2024-04-25T08:07:04.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:04.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2024-04-25T08:07:05.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2024-04-25T08:07:05.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2024-04-25T08:07:05.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2024-04-25T08:07:05.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2024-04-25T08:07:05.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2024-04-25T08:07:05.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2024-04-25T08:07:05.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2024-04-25T08:07:05.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2024-04-25T08:07:05.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2024-04-25T08:07:05.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2024-04-25T08:07:05.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:05.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2024-04-25T08:07:06.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2024-04-25T08:07:06.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2024-04-25T08:07:06.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2024-04-25T08:07:06.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2024-04-25T08:07:06.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2024-04-25T08:07:06.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2024-04-25T08:07:06.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2024-04-25T08:07:06.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2024-04-25T08:07:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2024-04-25T08:07:06.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:06.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2024-04-25T08:07:07.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2024-04-25T08:07:07.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2024-04-25T08:07:07.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2024-04-25T08:07:07.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2024-04-25T08:07:07.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2024-04-25T08:07:07.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2024-04-25T08:07:07.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2024-04-25T08:07:07.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2024-04-25T08:07:07.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2024-04-25T08:07:07.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2024-04-25T08:07:07.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:07.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2024-04-25T08:07:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2024-04-25T08:07:08.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2024-04-25T08:07:08.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2024-04-25T08:07:08.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2024-04-25T08:07:08.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2024-04-25T08:07:08.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2024-04-25T08:07:08.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2024-04-25T08:07:08.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2024-04-25T08:07:08.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2024-04-25T08:07:08.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:08.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2024-04-25T08:07:09.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2024-04-25T08:07:09.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2024-04-25T08:07:09.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2024-04-25T08:07:09.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2024-04-25T08:07:09.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2024-04-25T08:07:09.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2024-04-25T08:07:09.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2024-04-25T08:07:09.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2024-04-25T08:07:09.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2024-04-25T08:07:09.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:09.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2024-04-25T08:07:10.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2024-04-25T08:07:10.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2024-04-25T08:07:10.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2024-04-25T08:07:10.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2024-04-25T08:07:10.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2024-04-25T08:07:10.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2024-04-25T08:07:10.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2024-04-25T08:07:10.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2024-04-25T08:07:10.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2024-04-25T08:07:10.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2024-04-25T08:07:10.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:10.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2024-04-25T08:07:11.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2024-04-25T08:07:11.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2024-04-25T08:07:11.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2024-04-25T08:07:11.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2024-04-25T08:07:11.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2024-04-25T08:07:11.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2024-04-25T08:07:11.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2024-04-25T08:07:11.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2024-04-25T08:07:11.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2024-04-25T08:07:11.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:11.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2024-04-25T08:07:12.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2024-04-25T08:07:12.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2024-04-25T08:07:12.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2024-04-25T08:07:12.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2024-04-25T08:07:12.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2024-04-25T08:07:12.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2024-04-25T08:07:12.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2024-04-25T08:07:12.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2024-04-25T08:07:12.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2024-04-25T08:07:12.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:12.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2024-04-25T08:07:13.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2024-04-25T08:07:13.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2024-04-25T08:07:13.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2024-04-25T08:07:13.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2024-04-25T08:07:13.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2024-04-25T08:07:13.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2024-04-25T08:07:13.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2024-04-25T08:07:13.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2024-04-25T08:07:13.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:13.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2024-04-25T08:07:14.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2024-04-25T08:07:14.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2024-04-25T08:07:14.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2024-04-25T08:07:14.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2024-04-25T08:07:14.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2024-04-25T08:07:14.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2024-04-25T08:07:14.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2024-04-25T08:07:14.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2024-04-25T08:07:14.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2024-04-25T08:07:14.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:14.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2024-04-25T08:07:15.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2024-04-25T08:07:15.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2024-04-25T08:07:15.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2024-04-25T08:07:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2024-04-25T08:07:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2024-04-25T08:07:15.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2024-04-25T08:07:15.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2024-04-25T08:07:15.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2024-04-25T08:07:15.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:15.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2024-04-25T08:07:16.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2024-04-25T08:07:16.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2024-04-25T08:07:16.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2024-04-25T08:07:16.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2024-04-25T08:07:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2024-04-25T08:07:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2024-04-25T08:07:16.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2024-04-25T08:07:16.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2024-04-25T08:07:16.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2024-04-25T08:07:16.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2024-04-25T08:07:16.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:16.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2024-04-25T08:07:17.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2024-04-25T08:07:17.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2024-04-25T08:07:17.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2024-04-25T08:07:17.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2024-04-25T08:07:17.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2024-04-25T08:07:17.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2024-04-25T08:07:17.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2024-04-25T08:07:17.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2024-04-25T08:07:17.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2024-04-25T08:07:17.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:17.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2024-04-25T08:07:18.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2024-04-25T08:07:18.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2024-04-25T08:07:18.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2024-04-25T08:07:18.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2024-04-25T08:07:18.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2024-04-25T08:07:18.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2024-04-25T08:07:18.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2024-04-25T08:07:18.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2024-04-25T08:07:18.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2024-04-25T08:07:18.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:07:18.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2024-04-25T08:07:19.092 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: expr 2 - 1 2024-04-25T08:07:19.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: seq 0 1 2024-04-25T08:07:19.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:07:19.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2024-04-25T08:07:19.244 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T08:07:19.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:07:19.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2024-04-25T08:07:19.431 INFO:tasks.workunit.client.0.smithi088.stdout:{} 2024-04-25T08:07:19.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: sleep 5 2024-04-25T08:07:24.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:07:24.850 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:07:24.851 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 600 0 0 0 0 3072000 0 0 600 0 600 active+clean 2024-04-25T08:06:02.845004+0000 23'600 23:617 [1] 1 [1] 1 0'0 2024-04-25T08:05:59.666756+0000 0'0 2024-04-25T08:05:59.666756+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:45:34.403699+0000 0 0 2024-04-25T08:07:24.851 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:07:24.851 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:07:24.851 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T08:07:24.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:07:24.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-04-25T08:07:24.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:130: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2024-04-25T08:07:25.692 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 2 2024-04-25T08:07:25.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:134: TEST_recovery_test_simple: wait_for_recovery_toofull 30 2024-04-25T08:07:25.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: local timeout=30 2024-04-25T08:07:25.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:78: wait_for_recovery_toofull: wait_for_state recovery_toofull 30 2024-04-25T08:07:25.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local state=recovery_toofull 2024-04-25T08:07:25.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local cur_in_state 2024-04-25T08:07:25.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: get_timeout_delays 30 5 2024-04-25T08:07:25.715 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:07:25.715 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:07:25.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:07:25.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:07:25.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:07:25.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: delays=('5' '10' '15') 2024-04-25T08:07:25.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -a delays 2024-04-25T08:07:25.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:55: wait_for_state: local -i loop=0 2024-04-25T08:07:25.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: flush_pg_stats 2024-04-25T08:07:25.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:07:25.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:07:26.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:07:26.356 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T08:07:26.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:07:26.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:26.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:07:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836518 2024-04-25T08:07:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836518 2024-04-25T08:07:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518' 2024-04-25T08:07:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:26.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:07:26.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672992 2024-04-25T08:07:26.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672992 2024-04-25T08:07:26.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518 1-42949672992' 2024-04-25T08:07:26.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:26.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836518 2024-04-25T08:07:26.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:26.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:07:26.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836518 2024-04-25T08:07:26.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:26.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836518 2024-04-25T08:07:26.734 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836518 2024-04-25T08:07:26.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-04-25T08:07:26.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:07:27.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836518 -lt 21474836518 2024-04-25T08:07:27.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:27.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672992 2024-04-25T08:07:27.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:27.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:07:27.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672992 2024-04-25T08:07:27.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:27.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672992 2024-04-25T08:07:27.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-04-25T08:07:27.150 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672992 2024-04-25T08:07:27.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:07:27.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672992 -lt 42949672992 2024-04-25T08:07:27.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: get_num_pgs 2024-04-25T08:07:27.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:27.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: test 1 == 0 2024-04-25T08:07:28.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2024-04-25T08:07:28.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2024-04-25T08:07:28.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2024-04-25T08:07:28.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2024-04-25T08:07:28.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2024-04-25T08:07:28.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2024-04-25T08:07:28.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2024-04-25T08:07:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=1 2024-04-25T08:07:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 1 -gt 0 2024-04-25T08:07:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: break 2024-04-25T08:07:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:72: wait_for_state: return 0 2024-04-25T08:07:28.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: ERRORS=0 2024-04-25T08:07:28.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:07:28.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: grep +recovery_toofull 2024-04-25T08:07:28.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: wc -l 2024-04-25T08:07:28.945 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T08:07:28.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2024-04-25T08:07:28.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:143: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:07:29.328 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:07:29.328 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 600 0 600 0 0 3072000 0 0 600 0 600 active+recovery_toofull+undersized+degraded+remapped 2024-04-25T08:07:26.613179+0000 23'600 27:628 [1,0] 1 [1] 1 0'0 2024-04-25T08:05:59.666756+0000 0'0 2024-04-25T08:05:59.666756+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:07:29.328 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:07:29.328 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:07:29.329 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T08:07:29.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ceph status 2024-04-25T08:07:29.862 INFO:tasks.workunit.client.0.smithi088.stdout: cluster: 2024-04-25T08:07:29.862 INFO:tasks.workunit.client.0.smithi088.stdout: id: 25760e63-e148-4ff9-90a2-51e52013dc4d 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: health: HEALTH_ERR 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: 1 full osd(s) 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: 1 pool(s) do not have an application enabled 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: 1 pool(s) full 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: OSD count 2 < osd_pool_default_size 3 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: services: 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: mon: 1 daemons, quorum a (age 3m) 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: mgr: x(active, since 3m) 2024-04-25T08:07:29.863 INFO:tasks.workunit.client.0.smithi088.stdout: osd: 2 osds: 2 up (since 94s), 2 in (since 2m); 1 remapped pgs 2024-04-25T08:07:29.864 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:07:29.864 INFO:tasks.workunit.client.0.smithi088.stdout: data: 2024-04-25T08:07:29.864 INFO:tasks.workunit.client.0.smithi088.stdout: pools: 1 pools, 1 pgs 2024-04-25T08:07:29.864 INFO:tasks.workunit.client.0.smithi088.stdout: objects: 600 objects, 2.9 MiB 2024-04-25T08:07:29.864 INFO:tasks.workunit.client.0.smithi088.stdout: usage: 2.9 MiB used, 4.1 MiB / 7.0 MiB avail 2024-04-25T08:07:29.864 INFO:tasks.workunit.client.0.smithi088.stdout: pgs: 600/1200 objects degraded (50.000%) 2024-04-25T08:07:29.864 INFO:tasks.workunit.client.0.smithi088.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2024-04-25T08:07:29.864 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:07:29.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: ceph status --format=json-pretty 2024-04-25T08:07:30.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2024-04-25T08:07:30.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2024-04-25T08:07:30.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: SEV=HEALTH_ERR 2024-04-25T08:07:30.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:148: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2024-04-25T08:07:30.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2024-04-25T08:07:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2024-04-25T08:07:30.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2024-04-25T08:07:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:153: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2024-04-25T08:07:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:157: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2024-04-25T08:07:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:159: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2024-04-25T08:07:30.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:07:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2024-04-25T08:07:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:166: TEST_recovery_test_simple: delete_pool test1 2024-04-25T08:07:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:07:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:07:31.163 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test1' does not exist 2024-04-25T08:07:31.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:168: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2024-04-25T08:07:31.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:07:31.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:07:31.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:07:31.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:07:31.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:07:32.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:07:32.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:37: run: teardown td/osd-recovery-space 2024-04-25T08:07:32.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:07:32.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:07:32.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:07:32.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:07:32.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:07:32.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:07:32.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:07:32.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:07:32.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:07:32.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:07:32.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:07:32.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:07:32.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:07:32.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:07:32.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:07:32.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:32.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:07:32.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:07:32.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:32.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:07:32.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:07:32.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:07:32.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:07:32.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:32.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:07:32.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258125 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-space 0 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:07:32.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:07:32.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:07:32.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:07:32.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:07:32.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:07:32.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:07:32.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:07:32.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:32.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:07:32.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:07:32.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:32.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:07:32.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:07:32.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:07:32.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:07:32.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:32.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.258125 2024-04-25T08:07:32.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.258125 2024-04-25T08:07:32.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:07:32.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:07:32.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:07:32.553 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:07:32.553 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:07:32.564 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2024-04-25T08:07:32.565 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2024-04-25T08:07:32.615 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-stats 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2024-04-25T08:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2024-04-25T08:07:32.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T08:07:32.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2024-04-25T08:07:32.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2024-04-25T08:07:32.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none ' 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2024-04-25T08:07:32.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_recovery_erasure_remapped 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_recovery_multi 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_recovery_replicated_out1 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_recovery_sizedown 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_recovery_sizeup 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_recovery_undersized' 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:07:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:07:32.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:07:32.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:07:32.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:07:32.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:07:32.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:07:32.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:07:32.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:07:32.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:07:32.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:07:32.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:07:32.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:07:32.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:07:32.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:07:32.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:07:32.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:32.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:07:32.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:07:32.630 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:32.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:07:32.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:07:32.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:07:32.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:07:32.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:32.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:07:32.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:07:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:07:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:07:32.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:07:32.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:07:32.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:32.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:07:32.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274961 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-04-25T08:07:32.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-04-25T08:07:32.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2024-04-25T08:07:32.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-04-25T08:07:32.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:07:32.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:07:32.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:07:32.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:07:32.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:07:32.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:07:33.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:07:33.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:33.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:33.159 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:33.159 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:33.159 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:07:33.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:07:33.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:07:33.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:07:33.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:07:33.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:07:33.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:07:33.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:07:33.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:07:33.406 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:07:33.407 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:07:33.409 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:07:33.409 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:07:33.409 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:33.410 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:07:33.410 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:07:33.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:07:33.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get fsid 2024-04-25T08:07:33.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:07:33.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:07:33.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:07:33.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:07:33.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:07:33.527 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:07:33.527 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:07:33.527 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:07:33.528 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:07:33.528 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:33.528 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:07:33.528 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:07:33.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:07:33.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get mon_host 2024-04-25T08:07:33.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-04-25T08:07:33.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:07:33.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:07:33.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:07:33.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:07:33.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:07:33.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:07:34.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:07:34.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:34.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:34.576 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:34.576 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:34.576 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:07:34.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:07:34.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:07:34.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:07:34.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-04-25T08:07:34.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:07:34.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:07:34.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:07:34.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:07:34.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:07:34.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:34.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:34.615 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:07:34.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:07:34.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:07:34.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:07:34.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=068b3ebd-6f6e-4b40-ac72-a30d3c164dac 2024-04-25T08:07:34.618 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 068b3ebd-6f6e-4b40-ac72-a30d3c164dac 2024-04-25T08:07:34.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 068b3ebd-6f6e-4b40-ac72-a30d3c164dac' 2024-04-25T08:07:34.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:07:34.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGDypmR0n5JRAAsJqIGurbRK/dvnpdBuDKqg== 2024-04-25T08:07:34.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGDypmR0n5JRAAsJqIGurbRK/dvnpdBuDKqg=="}' 2024-04-25T08:07:34.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 068b3ebd-6f6e-4b40-ac72-a30d3c164dac -i td/osd-recovery-stats/0/new.json 2024-04-25T08:07:34.982 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:07:34.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:07:34.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGDypmR0n5JRAAsJqIGurbRK/dvnpdBuDKqg== --osd-uuid 068b3ebd-6f6e-4b40-ac72-a30d3c164dac 2024-04-25T08:07:35.031 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:35.033+0000 7f791899f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:35.067 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:35.069+0000 7f791899f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:35.068 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:35.069+0000 7f791899f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:35.069 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:35.069+0000 7f791899f7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:07:35.069 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:35.069+0000 7f791899f7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:07:35.069 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:35.069+0000 7f791899f7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:07:35.107 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:35.109+0000 7f791899f7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:07:42.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:07:42.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:07:42.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:07:42.921 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:07:42.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:07:43.492 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:07:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:07:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:07:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:07:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:07:43.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:07:43.523 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:43.524+0000 7fbb1296f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:43.526 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:43.528+0000 7fbb1296f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:43.527 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:43.528+0000 7fbb1296f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:43.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:07:43.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:07:43.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:07:43.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:07:43.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:07:43.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:43.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:07:43.890 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:07:43.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:43.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:44.079 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:44.080+0000 7fbb1296f7c0 -1 Falling back to public interface 2024-04-25T08:07:44.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:45.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:45.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:45.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:07:45.356 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:07:45.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:45.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:45.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:46.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:46.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:46.755 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:07:46.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:07:46.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:46.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:46.880 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:07:46.880+0000 7fbb1296f7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:07:47.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:48.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:48.256 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:07:48.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:48.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:07:48.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:48.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:48.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:49.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:49.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:49.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:07:49.699 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:07:49.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:49.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:50.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:51.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:51.345 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:07:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:07:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:51.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:51.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:52.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:52.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:52.784 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:07:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:07:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:53.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:54.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:54.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:54.213 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:07:54.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:07:54.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:54.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:54.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:55.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:55.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:55.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:07:55.640 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:07:55.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:55.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:56.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:57.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:57.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:57.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:07:57.247 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:07:57.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:57.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:57.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:58.720 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:07:58.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:07:58.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:58.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:59.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:00.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:00.250 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:08:00.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:00.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:08:00.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:00.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:00.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:01.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:01.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:01.965 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:08:01.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:08:01.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:01.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:02.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:03.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:03.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:03.616 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:08:03.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:08:03.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:03.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:04.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:05.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:05.094 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:08:05.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:05.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:08:05.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:05.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:05.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:06.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:06.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:06.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:08:06.588 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:08:06.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:06.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:07.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:08.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:08.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:08.714 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:08:08.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:08:08.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:08.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:09.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:10.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:10.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:10.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:08:10.675 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:08:10.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:10.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:11.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:12.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:08:12.439 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:08:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:13.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:14.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:14.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:14.489 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:08:14.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:08:14.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:14.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:15.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:16.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:16.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:16.329 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:08:16.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:08:16.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:16.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:17.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:18.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:08:18.232 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:08:18.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:18.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:08:18.698 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2003048944,v1:127.0.0.1:6803/2003048944] [v2:127.0.0.1:6804/2003048944,v1:127.0.0.1:6805/2003048944] exists,up 068b3ebd-6f6e-4b40-ac72-a30d3c164dac 2024-04-25T08:08:18.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:08:18.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:08:18.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:08:18.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:08:18.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:08:18.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:08:18.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:08:18.702 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:08:18.702 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:08:18.702 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:08:18.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:08:18.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:08:18.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:08:18.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:08:18.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=45d45cf1-6f9a-45cd-b3fc-47a69ffab135 2024-04-25T08:08:18.704 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 45d45cf1-6f9a-45cd-b3fc-47a69ffab135 2024-04-25T08:08:18.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 45d45cf1-6f9a-45cd-b3fc-47a69ffab135' 2024-04-25T08:08:18.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:08:18.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByDypmKkJgKxAALJjRPcggIJt/FtX9BEnqTA== 2024-04-25T08:08:18.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByDypmKkJgKxAALJjRPcggIJt/FtX9BEnqTA=="}' 2024-04-25T08:08:18.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 45d45cf1-6f9a-45cd-b3fc-47a69ffab135 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:08:19.318 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:08:19.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:08:19.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByDypmKkJgKxAALJjRPcggIJt/FtX9BEnqTA== --osd-uuid 45d45cf1-6f9a-45cd-b3fc-47a69ffab135 2024-04-25T08:08:19.369 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:19.368+0000 7faab5c297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:08:19.371 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:19.372+0000 7faab5c297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:08:19.374 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:19.372+0000 7faab5c297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:08:19.374 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:19.372+0000 7faab5c297c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:08:19.374 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:19.372+0000 7faab5c297c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:08:19.374 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:19.372+0000 7faab5c297c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:08:19.433 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:19.432+0000 7faab5c297c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:08:27.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:08:27.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:08:27.576 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:08:27.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:08:27.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:08:28.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:08:28.182 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:08:28.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:08:28.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:08:28.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:08:28.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:08:28.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:28.211+0000 7fa8a6a4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:08:28.215 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:28.215+0000 7fa8a6a4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:08:28.216 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:28.215+0000 7fa8a6a4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:08:28.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:08:28.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:08:28.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:08:28.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:08:28.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:08:28.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:28.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:08:28.599 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:08:28.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:28.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:28.767 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:28.767+0000 7fa8a6a4d7c0 -1 Falling back to public interface 2024-04-25T08:08:28.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:30.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:30.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:30.000 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:08:30.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:08:30.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:30.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:30.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:31.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:08:31.418 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:08:31.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:31.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:31.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:08:31.727+0000 7fa8a6a4d7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:08:31.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:32.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:32.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:32.995 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:08:32.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:08:32.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:32.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:33.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:34.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:34.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:34.612 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:08:34.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:08:34.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:34.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:35.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:36.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:36.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:36.260 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:08:36.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:08:36.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:36.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:37.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:38.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:38.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:38.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:08:38.041 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:08:38.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:38.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:38.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:39.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:39.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:39.472 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:08:39.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:08:39.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:39.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:40.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:41.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:41.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:41.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:08:41.013 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:08:41.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:41.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:41.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:42.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:42.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:42.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:08:42.554 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:08:42.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:42.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:43.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:44.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:44.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:44.089 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:08:44.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:08:44.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:44.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:44.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:45.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:45.685 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:08:45.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:08:45.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:45.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:46.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:47.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:47.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:47.220 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:08:47.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:08:47.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:47.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:47.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:48.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:48.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:48.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:08:48.724 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:08:48.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:48.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:49.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:50.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:50.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:50.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:08:50.401 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:08:50.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:50.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:51.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:52.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:52.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:52.040 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:08:52.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:08:52.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:52.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:54.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:54.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:54.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:08:54.375 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:08:54.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:54.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:55.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:56.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:56.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:56.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:08:56.254 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:08:56.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:56.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:58.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:58.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:58.082 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:08:58.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:08:58.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:58.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:08:58.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:08:59.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:08:59.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:08:59.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:08:59.844 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:08:59.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:08:59.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:09:00.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:01.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:01.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:01.823 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:09:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:09:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:01.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:09:02.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:03.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:03.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:03.962 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:09:03.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:09:03.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:03.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:09:04.417 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2614366061,v1:127.0.0.1:6811/2614366061] [v2:127.0.0.1:6812/2614366061,v1:127.0.0.1:6813/2614366061] exists,up 45d45cf1-6f9a-45cd-b3fc-47a69ffab135 2024-04-25T08:09:04.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:09:04.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:09:04.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:09:04.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-04-25T08:09:04.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:09:04.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:09:04.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:09:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:09:04.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:09:04.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:09:04.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:09:04.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:09:04.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa229497-ceff-44ad-9861-f57fb661cc52 2024-04-25T08:09:04.424 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 aa229497-ceff-44ad-9861-f57fb661cc52 2024-04-25T08:09:04.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 aa229497-ceff-44ad-9861-f57fb661cc52' 2024-04-25T08:09:04.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:09:04.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgDypmaP+uGhAAX8QmIte8+Zfwlm/KFDOvOA== 2024-04-25T08:09:04.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgDypmaP+uGhAAX8QmIte8+Zfwlm/KFDOvOA=="}' 2024-04-25T08:09:04.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa229497-ceff-44ad-9861-f57fb661cc52 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:09:05.174 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:09:05.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:09:05.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgDypmaP+uGhAAX8QmIte8+Zfwlm/KFDOvOA== --osd-uuid aa229497-ceff-44ad-9861-f57fb661cc52 2024-04-25T08:09:05.223 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:05.223+0000 7fdfde9c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:05.225 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:05.223+0000 7fdfde9c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:05.226 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:05.227+0000 7fdfde9c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:05.227 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:05.227+0000 7fdfde9c67c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:09:05.227 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:05.227+0000 7fdfde9c67c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:09:05.227 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:05.227+0000 7fdfde9c67c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:09:05.299 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:05.299+0000 7fdfde9c67c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:09:13.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:09:13.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:09:13.536 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:09:13.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:09:13.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:09:14.150 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:09:14.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:09:14.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:09:14.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:09:14.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:09:14.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:09:14.183 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:14.182+0000 7f5887e507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:14.206 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:14.206+0000 7f5887e507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:14.208 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:14.206+0000 7f5887e507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:14.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:09:14.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:09:14.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:09:14.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:09:14.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:09:14.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:14.592 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:09:14.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:09:14.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:14.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:14.767 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:14.766+0000 7f5887e507c0 -1 Falling back to public interface 2024-04-25T08:09:14.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:15.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:09:15.984 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:09:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:16.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:17.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:09:17.407 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:09:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:17.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:17.647 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:17.646+0000 7f5887e507c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:09:17.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:18.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:18.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:18.917 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:09:18.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:09:18.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:18.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:19.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:20.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:20.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:20.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:09:20.379 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:09:20.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:20.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:20.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:21.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:21.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:21.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:09:21.970 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:09:21.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:21.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:23.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:23.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:23.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:09:23.540 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:09:23.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:23.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:24.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:25.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:25.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:25.090 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:09:25.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:09:25.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:25.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:25.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:26.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:26.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:26.574 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:09:26.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:09:26.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:26.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:27.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:28.093 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:09:28.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:09:28.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:28.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:28.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:29.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:29.746 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:09:29.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:29.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:09:29.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:29.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:30.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:31.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:31.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:31.438 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:09:31.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:09:31.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:31.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:31.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:32.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:32.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:32.966 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:09:32.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:09:32.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:32.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:33.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:34.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:34.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:34.422 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:09:34.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:09:34.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:34.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:34.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:35.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:35.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:35.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:09:35.984 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:09:35.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:35.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:36.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:37.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:37.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:37.547 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:09:37.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:09:37.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:37.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:38.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:39.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:39.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:39.177 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:09:39.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:09:39.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:39.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:40.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:41.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:41.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:41.015 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:09:41.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:09:41.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:41.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:42.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:43.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:43.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:43.517 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:09:43.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:09:43.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:43.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:44.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:45.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:45.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:45.609 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:09:45.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:09:45.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:45.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:47.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:48.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:48.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:48.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:09:48.097 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:09:48.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:48.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:49.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:50.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:50.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:50.033 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:09:50.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:09:50.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:50.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:51.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:09:52.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:09:52.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:09:52.026 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:09:52.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:09:52.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:09:52.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:09:52.450 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1883778082,v1:127.0.0.1:6819/1883778082] [v2:127.0.0.1:6820/1883778082,v1:127.0.0.1:6821/1883778082] exists,up aa229497-ceff-44ad-9861-f57fb661cc52 2024-04-25T08:09:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:09:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:09:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:09:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:09:52.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:09:52.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:09:52.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:09:52.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:09:52.454 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:09:52.454 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:09:52.454 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:09:52.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:09:52.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:09:52.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:09:52.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:09:52.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cfd4f552-0e1e-4db9-99a1-fafe5eadfe83 2024-04-25T08:09:52.457 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 cfd4f552-0e1e-4db9-99a1-fafe5eadfe83 2024-04-25T08:09:52.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 cfd4f552-0e1e-4db9-99a1-fafe5eadfe83' 2024-04-25T08:09:52.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:09:52.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQDypmaGhkHBAAcRnmq2OeLrLb9qGhHHAfNA== 2024-04-25T08:09:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQDypmaGhkHBAAcRnmq2OeLrLb9qGhHHAfNA=="}' 2024-04-25T08:09:52.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cfd4f552-0e1e-4db9-99a1-fafe5eadfe83 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:09:53.003 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:09:53.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:09:53.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQDypmaGhkHBAAcRnmq2OeLrLb9qGhHHAfNA== --osd-uuid cfd4f552-0e1e-4db9-99a1-fafe5eadfe83 2024-04-25T08:09:53.057 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:53.058+0000 7f653a2f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:53.060 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:53.062+0000 7f653a2f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:53.061 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:53.062+0000 7f653a2f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:09:53.061 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:53.062+0000 7f653a2f47c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:09:53.061 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:53.062+0000 7f653a2f47c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:09:53.061 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:53.062+0000 7f653a2f47c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:09:53.136 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:09:53.138+0000 7f653a2f47c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:10:01.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:10:01.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:10:01.173 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T08:10:01.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:10:01.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:10:01.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:10:01.778 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T08:10:01.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:10:01.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:10:01.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:10:01.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:10:01.803 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:01.805+0000 7f54ddee67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:01.806 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:01.809+0000 7f54ddee67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:01.807 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:01.809+0000 7f54ddee67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:02.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:10:02.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:10:02.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:10:02.187 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:10:02.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:10:02.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:10:02.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:02.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:10:02.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:02.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:02.355 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:02.357+0000 7f54ddee67c0 -1 Falling back to public interface 2024-04-25T08:10:02.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:03.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:10:03.641 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:10:03.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:03.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:04.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:05.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:05.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:05.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:10:05.055 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:10:05.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:05.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:05.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:05.261+0000 7f54ddee67c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:10:05.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:06.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:06.478 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:10:06.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:10:06.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:06.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:07.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:08.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:08.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:08.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:10:08.092 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:10:08.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:08.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:08.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:09.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:09.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:09.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:10:09.702 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:10:09.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:09.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:10.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:10:11.183 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:10:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:12.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:13.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:13.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:13.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:10:13.312 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:10:13.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:13.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:14.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:15.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:15.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:15.169 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:10:15.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:10:15.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:15.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:16.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:17.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:17.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:17.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:10:17.032 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:10:17.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:17.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:17.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:10:18.765 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:10:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:18.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:19.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:20.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:20.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:20.183 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:10:20.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:10:20.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:20.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:20.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:21.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:21.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:21.840 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:10:21.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:10:21.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:21.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:22.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:23.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:23.444 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:10:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:10:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:23.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:10:24.928 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:10:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:24.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:25.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:26.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:26.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:26.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:10:26.426 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:10:26.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:26.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:27.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:28.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:28.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:28.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:10:28.154 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:10:28.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:28.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:29.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:30.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:30.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:30.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:10:30.007 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:10:30.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:30.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:30.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:31.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:31.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:31.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:10:31.830 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:10:31.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:31.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:32.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:33.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:33.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:33.928 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:10:33.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:10:33.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:33.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:34.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:36.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:36.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:36.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:10:36.000 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:10:36.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:36.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:37.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:38.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:38.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:38.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:10:38.156 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:10:38.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:38.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:10:38.633 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1085828484,v1:127.0.0.1:6827/1085828484] [v2:127.0.0.1:6828/1085828484,v1:127.0.0.1:6829/1085828484] exists,up cfd4f552-0e1e-4db9-99a1-fafe5eadfe83 2024-04-25T08:10:38.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:10:38.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:10:38.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:10:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:10:38.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:10:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:10:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:10:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:10:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:10:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:10:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:10:38.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:10:38.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=83a7eefe-6057-4824-b3d7-f41235fbe4fe 2024-04-25T08:10:38.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 83a7eefe-6057-4824-b3d7-f41235fbe4fe' 2024-04-25T08:10:38.640 INFO:tasks.workunit.client.0.smithi088.stdout:add osd4 83a7eefe-6057-4824-b3d7-f41235fbe4fe 2024-04-25T08:10:38.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:10:38.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+DypmNNRNJxAAK9pSinkLD6MN+F9cxTVN5g== 2024-04-25T08:10:38.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+DypmNNRNJxAAK9pSinkLD6MN+F9cxTVN5g=="}' 2024-04-25T08:10:38.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 83a7eefe-6057-4824-b3d7-f41235fbe4fe -i td/osd-recovery-stats/4/new.json 2024-04-25T08:10:39.166 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:10:39.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:10:39.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+DypmNNRNJxAAK9pSinkLD6MN+F9cxTVN5g== --osd-uuid 83a7eefe-6057-4824-b3d7-f41235fbe4fe 2024-04-25T08:10:39.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:39.213+0000 7f7a3cead7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:39.285 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:39.285+0000 7f7a3cead7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:39.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:39.289+0000 7f7a3cead7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:39.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:39.289+0000 7f7a3cead7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:10:39.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:39.289+0000 7f7a3cead7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:10:39.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:39.289+0000 7f7a3cead7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:10:39.397 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:39.401+0000 7f7a3cead7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:10:47.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:10:47.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:10:47.569 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd4 key to auth repository 2024-04-25T08:10:47.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:10:47.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:10:48.199 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.4 2024-04-25T08:10:48.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:10:48.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:10:48.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:10:48.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:10:48.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:10:48.233 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:48.232+0000 7f6623c507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:48.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:48.236+0000 7f6623c507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:48.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:48.236+0000 7f6623c507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:48.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:10:48.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:10:48.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:10:48.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:10:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:10:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:48.598 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:10:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:10:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:48.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:10:48.784 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:48.784+0000 7f6623c507c0 -1 Falling back to public interface 2024-04-25T08:10:49.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:50.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:50.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:50.059 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:10:50.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:10:50.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:50.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:10:50.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:51.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:51.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:51.468 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:10:51.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:10:51.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:51.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:10:51.706 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:10:51.704+0000 7f6623c507c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:10:51.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:52.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:52.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:52.967 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:10:52.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:10:52.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:52.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:10:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:54.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:54.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:54.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:10:54.683 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:10:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:10:55.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:56.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:56.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:56.302 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:10:56.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:10:56.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:56.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:10:56.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:57.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:57.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:57.843 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:10:57.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:10:57.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:57.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:10:58.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:59.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:59.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:59.447 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:10:59.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:10:59.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:59.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:00.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:01.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:01.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:01.089 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:11:01.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:11:01.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:01.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:01.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:02.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:02.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:02.589 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:11:02.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:11:02.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:02.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:03.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:04.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:04.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:04.217 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:11:04.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:11:04.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:04.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:04.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:05.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:05.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:05.817 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:11:05.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:11:05.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:05.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:06.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:07.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:07.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:07.460 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:11:07.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:11:07.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:07.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:08.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:09.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:09.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:09.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:11:09.077 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:11:09.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:09.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:09.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:10.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:10.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:10.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:11:10.608 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:11:10.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:10.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:11.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:12.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:12.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:12.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:11:12.486 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:11:12.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:12.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:13.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:14.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:14.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:14.772 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:11:14.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:11:14.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:14.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:15.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:16.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:16.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:16.773 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:11:16.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:11:16.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:16.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:18.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:19.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:19.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:19.059 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:11:19.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:11:19.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:19.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:19.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:20.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:20.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:20.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:11:20.867 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:11:20.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:20.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:21.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:22.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:22.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:22.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:11:22.828 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:11:22.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:22.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:23.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:24.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:24.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:24.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:11:24.568 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:11:24.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:24.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:24.998 INFO:tasks.workunit.client.0.smithi088.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4290796344,v1:127.0.0.1:6835/4290796344] [v2:127.0.0.1:6836/4290796344,v1:127.0.0.1:6837/4290796344] exists,up 83a7eefe-6057-4824-b3d7-f41235fbe4fe 2024-04-25T08:11:25.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:11:25.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:11:25.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:11:25.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-04-25T08:11:25.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:11:25.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:11:25.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:11:25.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:11:25.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:11:25.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc2c1790-bef1-4cca-b080-2de01fc25690 2024-04-25T08:11:25.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 cc2c1790-bef1-4cca-b080-2de01fc25690' 2024-04-25T08:11:25.007 INFO:tasks.workunit.client.0.smithi088.stdout:add osd5 cc2c1790-bef1-4cca-b080-2de01fc25690 2024-04-25T08:11:25.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:11:25.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtECpm92SFARAAITaIiBWvU0a3ILObc0Hgqg== 2024-04-25T08:11:25.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtECpm92SFARAAITaIiBWvU0a3ILObc0Hgqg=="}' 2024-04-25T08:11:25.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc2c1790-bef1-4cca-b080-2de01fc25690 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:11:25.856 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:11:25.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:11:25.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtECpm92SFARAAITaIiBWvU0a3ILObc0Hgqg== --osd-uuid cc2c1790-bef1-4cca-b080-2de01fc25690 2024-04-25T08:11:25.904 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:25.904+0000 7f8dcb7017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:25.945 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:25.944+0000 7f8dcb7017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:25.947 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:25.948+0000 7f8dcb7017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:25.947 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:25.948+0000 7f8dcb7017c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:11:25.947 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:25.948+0000 7f8dcb7017c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:11:25.948 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:25.948+0000 7f8dcb7017c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:11:26.053 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:26.052+0000 7f8dcb7017c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:11:34.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:11:34.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:11:34.609 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd5 key to auth repository 2024-04-25T08:11:34.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:11:34.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:11:35.188 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.5 2024-04-25T08:11:35.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:11:35.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:11:35.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:11:35.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:11:35.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:11:35.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:35.219+0000 7fef90d227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:35.222 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:35.219+0000 7fef90d227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:35.223 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:35.223+0000 7fef90d227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:35.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:11:35.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:11:35.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:11:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:11:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:11:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:35.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:11:35.608 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:11:35.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:35.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:35.780 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:35.779+0000 7fef90d227c0 -1 Falling back to public interface 2024-04-25T08:11:36.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:37.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:37.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:37.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:11:37.031 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:11:37.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:37.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:37.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:38.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:38.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:38.475 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:11:38.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:11:38.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:38.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:38.692 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:11:38.691+0000 7fef90d227c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:11:39.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:40.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:40.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:40.020 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:11:40.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:11:40.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:40.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:40.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:41.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:41.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:41.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:11:41.578 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:11:41.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:41.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:42.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:43.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:43.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:43.055 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:11:43.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:11:43.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:43.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:43.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:44.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:44.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:44.673 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:11:44.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:11:44.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:44.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:45.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:46.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:46.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:46.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:11:46.147 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:11:46.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:46.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:46.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:47.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:47.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:47.678 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:11:47.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:11:47.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:47.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:48.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:49.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:49.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:49.146 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:11:49.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:11:49.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:49.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:49.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:50.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:50.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:50.613 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:11:50.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:11:50.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:50.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:51.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:52.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:52.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:52.158 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:11:52.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:11:52.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:52.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:52.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:53.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:53.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:53.778 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:11:53.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:11:53.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:53.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:54.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:55.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:55.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:55.367 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:11:55.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:11:55.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:55.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:55.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:56.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:56.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:56.953 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:11:56.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:11:56.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:56.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:57.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:58.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:58.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:58.369 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:11:58.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:11:58.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:58.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:11:59.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:12:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:12:00.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:12:00.649 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:12:00.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:12:00.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:12:00.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:12:01.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:12:02.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:12:02.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:12:02.496 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:12:02.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:12:02.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:12:02.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:12:03.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:12:04.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:12:04.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:12:04.319 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:12:04.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:12:04.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:12:04.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:12:05.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:12:06.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:12:06.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:12:06.283 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:12:06.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:12:06.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:12:06.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:12:07.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:12:08.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:12:08.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:12:08.251 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:12:08.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:12:08.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:12:08.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:12:09.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:12:10.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:12:10.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:12:10.252 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:12:10.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:12:10.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:12:10.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:12:11.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:12:12.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:12:12.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:12:12.282 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:12:12.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:12:12.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:12:12.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:12:12.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:12:13.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:12:13.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:12:13.787 INFO:tasks.workunit.client.0.smithi088.stdout:23 2024-04-25T08:12:13.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-04-25T08:12:13.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:12:13.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:12:14.239 INFO:tasks.workunit.client.0.smithi088.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/944250345,v1:127.0.0.1:6843/944250345] [v2:127.0.0.1:6844/944250345,v1:127.0.0.1:6845/944250345] exists,up cc2c1790-bef1-4cca-b080-2de01fc25690 2024-04-25T08:12:14.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:12:14.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:12:14.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:12:14.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2024-04-25T08:12:14.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T08:12:14.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2024-04-25T08:12:14.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T08:12:16.119 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T08:12:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:12:17.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-04-25T08:12:17.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:12:17.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:12:17.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:12:17.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:12:17.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:12:17.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:12:17.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:12:17.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:12:17.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:12:17.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:12:17.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:12:17.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:12:17.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:12:17.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:12:17.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:12:17.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:12:17.699 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:12:17.699 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:12:17.700 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:12:17.700 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:12:17.700 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:12:17.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:12:17.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:17.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:12:17.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836541 2024-04-25T08:12:17.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836541 2024-04-25T08:12:17.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836541' 2024-04-25T08:12:17.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:17.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:12:18.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705709 2024-04-25T08:12:18.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705709 2024-04-25T08:12:18.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836541 1-38654705709' 2024-04-25T08:12:18.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:18.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:12:18.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542188 2024-04-25T08:12:18.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542188 2024-04-25T08:12:18.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836541 1-38654705709 2-60129542188' 2024-04-25T08:12:18.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:18.203 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:12:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378652 2024-04-25T08:12:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378652 2024-04-25T08:12:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836541 1-38654705709 2-60129542188 3-81604378652' 2024-04-25T08:12:18.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:18.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:12:18.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215125 2024-04-25T08:12:18.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215125 2024-04-25T08:12:18.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836541 1-38654705709 2-60129542188 3-81604378652 4-103079215125' 2024-04-25T08:12:18.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:18.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:12:18.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051594 2024-04-25T08:12:18.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051594 2024-04-25T08:12:18.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836541 1-38654705709 2-60129542188 3-81604378652 4-103079215125 5-124554051594' 2024-04-25T08:12:18.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:18.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:18.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836541 2024-04-25T08:12:18.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:12:18.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836541 2024-04-25T08:12:18.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:18.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836541 2024-04-25T08:12:18.655 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836541 2024-04-25T08:12:18.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-04-25T08:12:18.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:12:19.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836539 -lt 21474836541 2024-04-25T08:12:19.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:12:20.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:12:20.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:12:20.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836541 -lt 21474836541 2024-04-25T08:12:20.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:20.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:20.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705709 2024-04-25T08:12:20.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:12:20.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705709 2024-04-25T08:12:20.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:20.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705709 2024-04-25T08:12:20.475 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 38654705709 2024-04-25T08:12:20.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705709' 2024-04-25T08:12:20.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:12:20.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705709 -lt 38654705709 2024-04-25T08:12:20.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:20.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542188 2024-04-25T08:12:20.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:20.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:12:20.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542188 2024-04-25T08:12:20.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:20.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542188 2024-04-25T08:12:20.867 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 60129542188 2024-04-25T08:12:20.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542188' 2024-04-25T08:12:20.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:12:21.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542188 -lt 60129542188 2024-04-25T08:12:21.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:21.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378652 2024-04-25T08:12:21.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:21.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:12:21.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378652 2024-04-25T08:12:21.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:21.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378652 2024-04-25T08:12:21.321 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 81604378652 2024-04-25T08:12:21.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378652' 2024-04-25T08:12:21.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:12:21.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378652 -lt 81604378652 2024-04-25T08:12:21.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:21.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215125 2024-04-25T08:12:21.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:21.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:12:21.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215125 2024-04-25T08:12:21.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:21.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215125 2024-04-25T08:12:21.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215125' 2024-04-25T08:12:21.726 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 103079215125 2024-04-25T08:12:21.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:12:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215126 -lt 103079215125 2024-04-25T08:12:22.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:22.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:22.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-124554051594 2024-04-25T08:12:22.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:12:22.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-124554051594 2024-04-25T08:12:22.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051594 2024-04-25T08:12:22.193 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 124554051594 2024-04-25T08:12:22.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 124554051594' 2024-04-25T08:12:22.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:12:22.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051594 -lt 124554051594 2024-04-25T08:12:22.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:12:22.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:22.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:23.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:12:23.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:23.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:23.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:23.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:23.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:23.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:23.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:23.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:12:23.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:23.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:23.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:24.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:12:24.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:12:24.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:12:24.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-04-25T08:12:24.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:24.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-04-25T08:12:24.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:24.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-04-25T08:12:24.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:24.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-04-25T08:12:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-04-25T08:12:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-04-25T08:12:25.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:25.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-04-25T08:12:25.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:25.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-04-25T08:12:26.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:26.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-04-25T08:12:26.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:26.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-04-25T08:12:26.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:26.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-04-25T08:12:26.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:26.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-04-25T08:12:27.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:27.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-04-25T08:12:27.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:27.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-04-25T08:12:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-04-25T08:12:28.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:28.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-04-25T08:12:28.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:28.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-04-25T08:12:28.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:28.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-04-25T08:12:28.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:28.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-04-25T08:12:29.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:29.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-04-25T08:12:29.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:29.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-04-25T08:12:29.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:29.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-04-25T08:12:29.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:29.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-04-25T08:12:30.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:30.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-04-25T08:12:30.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:30.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-04-25T08:12:30.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:30.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-04-25T08:12:31.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:31.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-04-25T08:12:31.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:31.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-04-25T08:12:31.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:31.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-04-25T08:12:31.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:31.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-04-25T08:12:32.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:32.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-04-25T08:12:32.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:32.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-04-25T08:12:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-04-25T08:12:32.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:32.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-04-25T08:12:33.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:33.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-04-25T08:12:33.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:33.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-04-25T08:12:33.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:33.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-04-25T08:12:33.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:33.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-04-25T08:12:34.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:34.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-04-25T08:12:34.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:34.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-04-25T08:12:34.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:34.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-04-25T08:12:35.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:35.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-04-25T08:12:35.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:35.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-04-25T08:12:35.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:35.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-04-25T08:12:35.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:35.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-04-25T08:12:36.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:36.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-04-25T08:12:36.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:36.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-04-25T08:12:36.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:36.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-04-25T08:12:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-04-25T08:12:37.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:37.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-04-25T08:12:37.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:37.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-04-25T08:12:37.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:37.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-04-25T08:12:38.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:38.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-04-25T08:12:38.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:38.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-04-25T08:12:38.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:38.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-04-25T08:12:38.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:38.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-04-25T08:12:39.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:39.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-04-25T08:12:39.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:39.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-04-25T08:12:39.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:39.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-04-25T08:12:40.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:40.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-04-25T08:12:40.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:40.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-04-25T08:12:40.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:40.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-04-25T08:12:40.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:40.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-04-25T08:12:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:41.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-04-25T08:12:41.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:41.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-04-25T08:12:41.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:41.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-04-25T08:12:42.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:42.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-04-25T08:12:42.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:42.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-04-25T08:12:42.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:42.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-04-25T08:12:43.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:43.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-04-25T08:12:43.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:43.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-04-25T08:12:43.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:43.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-04-25T08:12:44.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:44.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-04-25T08:12:44.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:44.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-04-25T08:12:44.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:44.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-04-25T08:12:44.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:44.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-04-25T08:12:45.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:45.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-04-25T08:12:45.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:45.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-04-25T08:12:45.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:45.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-04-25T08:12:46.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:46.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-04-25T08:12:46.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:46.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-04-25T08:12:46.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:46.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-04-25T08:12:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:46.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-04-25T08:12:47.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:47.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-04-25T08:12:47.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:47.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-04-25T08:12:47.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:47.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-04-25T08:12:47.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:47.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-04-25T08:12:48.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:48.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-04-25T08:12:48.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:48.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-04-25T08:12:48.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:48.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-04-25T08:12:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:48.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-04-25T08:12:49.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:49.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-04-25T08:12:49.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:49.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-04-25T08:12:49.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:49.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-04-25T08:12:50.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:50.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-04-25T08:12:50.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:50.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-04-25T08:12:50.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:50.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-04-25T08:12:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-04-25T08:12:51.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:51.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-04-25T08:12:51.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:51.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-04-25T08:12:52.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:52.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-04-25T08:12:52.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:52.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-04-25T08:12:52.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:52.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-04-25T08:12:52.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:52.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-04-25T08:12:53.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:53.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-04-25T08:12:53.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:53.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-04-25T08:12:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-04-25T08:12:53.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:53.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-04-25T08:12:54.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:54.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-04-25T08:12:54.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:54.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-04-25T08:12:54.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:54.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-04-25T08:12:55.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:55.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-04-25T08:12:55.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:55.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-04-25T08:12:55.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:55.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-04-25T08:12:56.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:56.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-04-25T08:12:56.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:56.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-04-25T08:12:56.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:56.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-04-25T08:12:56.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:56.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-04-25T08:12:57.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:57.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-04-25T08:12:57.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:57.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-04-25T08:12:57.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:57.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-04-25T08:12:58.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:58.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-04-25T08:12:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-04-25T08:12:58.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:58.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-04-25T08:12:58.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:58.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-04-25T08:12:59.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:59.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-04-25T08:12:59.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:59.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-04-25T08:12:59.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:12:59.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-04-25T08:13:00.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:00.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-04-25T08:13:00.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:00.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-04-25T08:13:00.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:00.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-04-25T08:13:00.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:00.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-04-25T08:13:01.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:01.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-04-25T08:13:01.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:01.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-04-25T08:13:01.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:01.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-04-25T08:13:02.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:02.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-04-25T08:13:02.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:02.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-04-25T08:13:02.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:02.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-04-25T08:13:02.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:02.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-04-25T08:13:03.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:03.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-04-25T08:13:03.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:03.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-04-25T08:13:03.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:03.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-04-25T08:13:04.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:04.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-04-25T08:13:04.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:04.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-04-25T08:13:04.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:04.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-04-25T08:13:04.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:04.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-04-25T08:13:05.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:05.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-04-25T08:13:05.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:05.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-04-25T08:13:05.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:05.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-04-25T08:13:06.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:06.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-04-25T08:13:06.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:06.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-04-25T08:13:06.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:06.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-04-25T08:13:06.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:06.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-04-25T08:13:07.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:07.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-04-25T08:13:07.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:07.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-04-25T08:13:07.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:07.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-04-25T08:13:08.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:08.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-04-25T08:13:08.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:08.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-04-25T08:13:08.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:08.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-04-25T08:13:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-04-25T08:13:09.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:09.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-04-25T08:13:09.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:09.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-04-25T08:13:09.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:09.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-04-25T08:13:10.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:10.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-04-25T08:13:10.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:10.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-04-25T08:13:10.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:10.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-04-25T08:13:10.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:10.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-04-25T08:13:11.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:11.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-04-25T08:13:11.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:11.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-04-25T08:13:11.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:11.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-04-25T08:13:12.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:12.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-04-25T08:13:12.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:12.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-04-25T08:13:12.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:12.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-04-25T08:13:12.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:12.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-04-25T08:13:13.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:13.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-04-25T08:13:13.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:13.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-04-25T08:13:14.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:14.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-04-25T08:13:14.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:14.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-04-25T08:13:14.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:14.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-04-25T08:13:15.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:15.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-04-25T08:13:15.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:15.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-04-25T08:13:15.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:15.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-04-25T08:13:15.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:15.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-04-25T08:13:16.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:16.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-04-25T08:13:16.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:16.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-04-25T08:13:16.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:16.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-04-25T08:13:17.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:17.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-04-25T08:13:17.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:17.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-04-25T08:13:17.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:17.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-04-25T08:13:17.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:17.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-04-25T08:13:18.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:18.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-04-25T08:13:18.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:18.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-04-25T08:13:18.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:18.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-04-25T08:13:19.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:19.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-04-25T08:13:19.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:19.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-04-25T08:13:19.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:19.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-04-25T08:13:20.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:20.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-04-25T08:13:20.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:20.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-04-25T08:13:20.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:20.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-04-25T08:13:20.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:20.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-04-25T08:13:21.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:13:21.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-04-25T08:13:21.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-04-25T08:13:21.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:13:21.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:13:21.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:13:21.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:13:21.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-04-25T08:13:21.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-04-25T08:13:21.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:13:21.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:13:21.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:13:21.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:13:22.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-04-25T08:13:22.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-04-25T08:13:22.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:13:22.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:13:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:13:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:13:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:13:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:13:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:13:22.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:13:22.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:13:22.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:13:23.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-04-25T08:13:23.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-04-25T08:13:24.167 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is set 2024-04-25T08:13:24.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:13:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 276042 2024-04-25T08:13:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-04-25T08:13:24.869 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T08:13:24.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-04-25T08:13:26.242 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T08:13:26.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-04-25T08:13:27.423 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is unset 2024-04-25T08:13:27.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-04-25T08:13:27.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:13:27.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:13:27.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:13:27.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:13:28.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:13:28.465 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:13:28.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-04-25T08:13:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-04-25T08:13:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:13:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:13:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:13:30.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:13:30.484 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:13:30.484 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:13:30.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:13:30.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:13:30.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:13:30.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:13:30.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:13:30.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:13:30.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:13:30.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:13:30.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:13:31.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:13:31.039 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:13:31.039 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:13:31.039 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:13:31.039 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:13:31.039 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:13:31.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:13:31.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:13:31.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:13:31.273 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:13:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:13:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:13:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:13:31.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:13:31.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:13:31.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705726 2024-04-25T08:13:31.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705726 2024-04-25T08:13:31.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-38654705726' 2024-04-25T08:13:31.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:13:31.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:13:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542204 2024-04-25T08:13:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542204 2024-04-25T08:13:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-38654705726 2-60129542204' 2024-04-25T08:13:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:13:31.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:13:31.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378668 2024-04-25T08:13:31.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378668 2024-04-25T08:13:31.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-38654705726 2-60129542204 3-81604378668' 2024-04-25T08:13:31.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:13:31.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:13:32.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215142 2024-04-25T08:13:32.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215142 2024-04-25T08:13:32.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-38654705726 2-60129542204 3-81604378668 4-103079215142' 2024-04-25T08:13:32.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:13:32.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:13:32.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051610 2024-04-25T08:13:32.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051610 2024-04-25T08:13:32.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-38654705726 2-60129542204 3-81604378668 4-103079215142 5-124554051610' 2024-04-25T08:13:32.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:13:32.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705726 2024-04-25T08:13:32.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:13:32.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:13:32.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705726 2024-04-25T08:13:32.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:13:32.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705726 2024-04-25T08:13:32.221 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 38654705726 2024-04-25T08:13:32.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705726' 2024-04-25T08:13:32.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:13:32.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705724 -lt 38654705726 2024-04-25T08:13:32.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:13:33.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:13:33.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:13:34.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705726 -lt 38654705726 2024-04-25T08:13:34.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:13:34.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542204 2024-04-25T08:13:34.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:13:34.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:13:34.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542204 2024-04-25T08:13:34.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:13:34.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542204 2024-04-25T08:13:34.154 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 60129542204 2024-04-25T08:13:34.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542204' 2024-04-25T08:13:34.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:13:34.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542205 -lt 60129542204 2024-04-25T08:13:34.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:13:34.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378668 2024-04-25T08:13:34.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:13:34.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:13:34.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378668 2024-04-25T08:13:34.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:13:34.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378668 2024-04-25T08:13:34.708 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 81604378668 2024-04-25T08:13:34.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378668' 2024-04-25T08:13:34.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:13:35.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378669 -lt 81604378668 2024-04-25T08:13:35.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:13:35.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215142 2024-04-25T08:13:35.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:13:35.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:13:35.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215142 2024-04-25T08:13:35.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:13:35.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215142 2024-04-25T08:13:35.145 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 103079215142 2024-04-25T08:13:35.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215142' 2024-04-25T08:13:35.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:13:35.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215142 -lt 103079215142 2024-04-25T08:13:35.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:13:35.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:13:35.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-124554051610 2024-04-25T08:13:35.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:13:35.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-124554051610 2024-04-25T08:13:35.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:13:35.696 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 124554051610 2024-04-25T08:13:35.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051610 2024-04-25T08:13:35.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 124554051610' 2024-04-25T08:13:35.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:13:36.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051611 -lt 124554051610 2024-04-25T08:13:36.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:13:36.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:36.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:13:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:36.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:36.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:36.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:36.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:36.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:36.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:37.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:37.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:37.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:37.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:37.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:37.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:13:37.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:13:37.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:13:37.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:37.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:37.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:37.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:37.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:37.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:37.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:37.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:37.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:37.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:38.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:38.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:38.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:38.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:38.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:38.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:38.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:38.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:38.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:38.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:38.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:38.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:38.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:39.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:13:39.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:13:39.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:39.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:39.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:39.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:39.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:39.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:39.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:39.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:39.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:39.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:39.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:39.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:39.978 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:39.978 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:39.978 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:40.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:40.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:40.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:41.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:13:41.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:13:41.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:41.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:41.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:41.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:41.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:41.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:41.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:41.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:41.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:41.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:41.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:41.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:41.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:41.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:41.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:42.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:42.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:42.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:42.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:42.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:42.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:42.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:42.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:42.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:43.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:13:43.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:13:43.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:43.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:43.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:43.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:43.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:43.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:43.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:43.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:43.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:43.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:43.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:43.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:43.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:43.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:43.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:44.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:44.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:44.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:44.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:44.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:44.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:44.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:44.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:44.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:13:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:13:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:44.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:45.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:45.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:45.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:45.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:45.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:45.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:45.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:45.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:45.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:45.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:45.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:45.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:46.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:46.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:46.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:46.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:46.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:46.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:46.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:46.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:46.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:46.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:13:46.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:13:46.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:46.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:46.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:46.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:46.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:46.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:46.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:46.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:46.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:46.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:46.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:47.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:47.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:47.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:47.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:47.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:47.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:47.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:47.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:47.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:47.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:47.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:47.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:47.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:48.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:13:48.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:13:48.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:48.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:48.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:48.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:48.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:48.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:48.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:48.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:48.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:48.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:48.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:48.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:48.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:48.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:48.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:49.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:49.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:49.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:50.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:13:50.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:13:50.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:50.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:50.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:50.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:50.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:50.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:50.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:50.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:50.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:50.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:50.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:50.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:50.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:50.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:50.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:51.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:51.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:51.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:51.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:51.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:51.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:51.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:51.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:51.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:51.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:13:51.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:13:51.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:51.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:51.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:51.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:51.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:51.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:51.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:51.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:51.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:51.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:51.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:52.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:52.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:52.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:52.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:52.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:52.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:52.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:52.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:52.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:52.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:52.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:52.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:52.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:13:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:13:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:53.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:53.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:53.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:53.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:53.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:53.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:53.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:53.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:53.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:53.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:53.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:53.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:54.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:13:54.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:54.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:54.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:54.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:54.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:54.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:54.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:54.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:55.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:13:55.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:13:55.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:55.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:55.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:55.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:55.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:55.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:55.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:55.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:55.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:55.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:55.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:55.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:13:55.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:55.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:55.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:56.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:13:56.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:13:56.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-04-25T08:13:56.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-04-25T08:13:56.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:13:56.085 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:13:56.085 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:13:56.085 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:13:56.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2024-04-25T08:13:56.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:13:56.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:13:56.106 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:13:56.106 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:13:56.106 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-04-25T08:13:56.107 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:13:56.107 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:13:56.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:13:56.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:13:56.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:13:56.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:13:56.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:13:56.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:13:56.115 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:13:56.115 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:13:56.115 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:13:56.115 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:13:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:13:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:13:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:13:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:13:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:13:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:13:56.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:13:56.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:13:56.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:13:56.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-04-25T08:13:56.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:13:56.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:13:56.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:13:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:13:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:13:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:13:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:13:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:13:56.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:13:56.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:13:56.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:13:56.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:13:56.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-04-25T08:13:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:13:56.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:13:56.670 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T08:13:56.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-04-25T08:13:56.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:13:56.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:13:56.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:13:56.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:13:56.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:14:02.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:14:02.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:14:02.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:14:02.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:14:02.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:14:02.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:14:02.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:14:02.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:14:02.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:14:02.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:14:02.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:14:02.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:14:02.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:14:02.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:14:02.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:14:02.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:14:02.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:02.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:02.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:14:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:14:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:14:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:14:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:14:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:14:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:14:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:14:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:14:02.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:14:02.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:14:02.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:14:02.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:14:02.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:14:02.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:14:02.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:14:02.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:14:02.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:14:02.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:14:02.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:14:02.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:14:02.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:14:02.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:14:02.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:14:02.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:14:02.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:14:02.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:14:02.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:14:02.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:14:02.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:14:02.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:02.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:02.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:14:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:14:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:14:02.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:14:02.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:14:02.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:02.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:02.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274961 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:14:02.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:14:02.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:14:02.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:14:02.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:14:02.620 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:14:02.620 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:02.620 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:02.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:14:02.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:14:02.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:14:02.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:14:02.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:14:02.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:14:02.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:14:02.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:14:02.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:14:02.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:14:02.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:14:02.876 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:02.876 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:02.876 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:02.877 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:14:02.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:14:02.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get fsid 2024-04-25T08:14:02.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:14:02.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:14:02.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:14:02.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:14:02.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:14:02.994 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:14:02.994 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:14:02.994 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:14:02.995 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:14:02.995 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:02.995 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:02.995 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:14:02.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:14:02.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get mon_host 2024-04-25T08:14:03.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2024-04-25T08:14:03.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:14:03.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:14:03.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:14:03.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:14:03.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:14:03.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:14:04.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:14:04.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:14:04.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:14:04.045 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:14:04.045 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:04.045 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:04.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:14:04.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:14:04.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:14:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2024-04-25T08:14:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:14:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:14:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:14:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:14:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:14:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:14:04.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:14:04.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:14:04.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:14:04.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bdc61aeb-b0fc-4860-89ab-8c7fa00603ff 2024-04-25T08:14:04.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bdc61aeb-b0fc-4860-89ab-8c7fa00603ff' 2024-04-25T08:14:04.085 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 bdc61aeb-b0fc-4860-89ab-8c7fa00603ff 2024-04-25T08:14:04.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:14:04.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMECpmjdI3BhAA24c+3wrEd3GD1NmQ8fsxTQ== 2024-04-25T08:14:04.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMECpmjdI3BhAA24c+3wrEd3GD1NmQ8fsxTQ=="}' 2024-04-25T08:14:04.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bdc61aeb-b0fc-4860-89ab-8c7fa00603ff -i td/osd-recovery-stats/0/new.json 2024-04-25T08:14:04.459 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:14:04.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:14:04.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMECpmjdI3BhAA24c+3wrEd3GD1NmQ8fsxTQ== --osd-uuid bdc61aeb-b0fc-4860-89ab-8c7fa00603ff 2024-04-25T08:14:04.511 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:04.512+0000 7ff2a6cec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:04.544 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:04.544+0000 7ff2a6cec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:04.546 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:04.548+0000 7ff2a6cec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:04.546 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:04.548+0000 7ff2a6cec7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:14:04.547 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:04.548+0000 7ff2a6cec7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:14:04.547 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:04.548+0000 7ff2a6cec7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:14:04.584 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:04.584+0000 7ff2a6cec7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:14:12.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:14:12.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:14:12.397 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:14:12.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:14:12.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:14:12.984 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:14:12.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:14:12.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:14:12.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:14:12.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:14:12.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:14:13.015 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:13.016+0000 7fad795847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:13.017 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:13.016+0000 7fad795847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:13.019 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:13.020+0000 7fad795847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:13.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:14:13.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:14:13.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:14:13.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:14:13.387 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:14:13.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:14:13.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:13.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:14:13.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:13.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:13.564 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:13.564+0000 7fad795847c0 -1 Falling back to public interface 2024-04-25T08:14:13.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:14.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:14.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:14.783 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:14:14.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:14:14.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:14.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:15.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:16.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:16.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:16.214 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:14:16.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:14:16.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:16.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:16.409 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:16.408+0000 7fad795847c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:14:16.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:17.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:17.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:17.676 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:14:17.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:14:17.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:17.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:18.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:19.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:19.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:19.163 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:14:19.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:14:19.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:19.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:19.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:20.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:20.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:20.632 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:14:20.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:14:20.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:20.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:21.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:22.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:22.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:22.070 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:14:22.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:14:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:22.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:22.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:23.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:23.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:23.670 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:14:23.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:14:23.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:23.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:24.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:25.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:25.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:25.163 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:14:25.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:14:25.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:25.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:25.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:26.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:26.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:26.610 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:14:26.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:14:26.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:26.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:27.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:28.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:28.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:28.044 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:14:28.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:14:28.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:28.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:28.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:29.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:29.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:29.619 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:14:29.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:14:29.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:29.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:30.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:31.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:31.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:31.195 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:14:31.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:14:31.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:31.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:31.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:32.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:32.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:32.725 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:14:32.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:14:32.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:32.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:33.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:34.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:34.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:34.311 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:14:34.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:14:34.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:34.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:34.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:35.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:35.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:35.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:14:35.853 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:14:35.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:35.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:36.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:37.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:37.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:37.318 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:14:37.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:14:37.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:37.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:38.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:39.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:39.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:39.019 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:14:39.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:14:39.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:39.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:40.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:41.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:41.473 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:14:41.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:14:41.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:41.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:42.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:43.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:43.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:43.428 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:14:43.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:14:43.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:43.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:44.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:45.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:45.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:45.743 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:14:45.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:14:45.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:45.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:46.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:47.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:47.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:47.682 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:14:47.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:14:47.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:47.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:48.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:14:49.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:14:49.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:49.249 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:14:49.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:14:49.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:49.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:14:49.674 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1507582811,v1:127.0.0.1:6803/1507582811] [v2:127.0.0.1:6804/1507582811,v1:127.0.0.1:6805/1507582811] exists,up bdc61aeb-b0fc-4860-89ab-8c7fa00603ff 2024-04-25T08:14:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:14:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:14:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:14:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2024-04-25T08:14:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:14:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:14:49.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:14:49.677 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:14:49.678 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:14:49.678 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:14:49.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:14:49.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:14:49.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:14:49.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:14:49.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0bd636e1-b8bf-4cf5-b78e-735ee32cdf7f 2024-04-25T08:14:49.680 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 0bd636e1-b8bf-4cf5-b78e-735ee32cdf7f 2024-04-25T08:14:49.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0bd636e1-b8bf-4cf5-b78e-735ee32cdf7f' 2024-04-25T08:14:49.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:14:49.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5ECpmDOQBKhAAvYdZxe7e5IoAC133qKYYew== 2024-04-25T08:14:49.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5ECpmDOQBKhAAvYdZxe7e5IoAC133qKYYew=="}' 2024-04-25T08:14:49.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0bd636e1-b8bf-4cf5-b78e-735ee32cdf7f -i td/osd-recovery-stats/1/new.json 2024-04-25T08:14:50.286 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:14:50.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:14:50.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5ECpmDOQBKhAAvYdZxe7e5IoAC133qKYYew== --osd-uuid 0bd636e1-b8bf-4cf5-b78e-735ee32cdf7f 2024-04-25T08:14:50.338 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:50.340+0000 7f522d81a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:50.341 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:50.340+0000 7f522d81a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:50.342 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:50.340+0000 7f522d81a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:50.343 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:50.344+0000 7f522d81a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:14:50.343 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:50.344+0000 7f522d81a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:14:50.343 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:50.344+0000 7f522d81a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:14:50.411 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:50.412+0000 7f522d81a7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:14:58.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:14:58.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:14:58.616 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:14:58.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:14:58.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:14:59.193 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:14:59.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:14:59.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:14:59.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:14:59.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:14:59.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:14:59.219 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:59.219+0000 7f3e3d4647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:59.221 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:59.219+0000 7f3e3d4647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:59.222 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:59.223+0000 7f3e3d4647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:14:59.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:14:59.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:14:59.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:14:59.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:14:59.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:14:59.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:14:59.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:14:59.601 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:14:59.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:14:59.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:14:59.759 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:14:59.759+0000 7f3e3d4647c0 -1 Falling back to public interface 2024-04-25T08:15:00.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:01.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:01.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:01.024 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:15:01.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:15:01.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:01.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:01.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:02.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:02.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:02.474 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:15:02.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:15:02.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:02.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:02.661 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:02.659+0000 7f3e3d4647c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:15:02.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:03.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:03.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:03.902 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:15:03.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:15:03.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:03.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:04.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:05.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:05.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:05.405 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:15:05.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:15:05.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:05.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:06.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:07.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:07.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:07.163 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:15:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:15:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:07.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:07.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:08.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:08.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:08.820 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:15:08.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:15:08.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:08.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:09.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:10.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:10.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:10.603 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:15:10.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:15:10.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:10.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:11.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:12.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:12.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:12.187 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:15:12.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:15:12.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:12.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:12.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:13.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:13.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:13.740 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:15:13.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:15:13.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:13.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:14.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:15.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:15.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:15.289 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:15:15.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:15:15.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:15.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:15.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:16.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:16.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:16.743 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:15:16.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:15:16.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:16.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:17.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:18.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:18.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:18.539 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:15:18.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:15:18.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:18.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:19.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:20.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:20.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:20.300 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:15:20.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:15:20.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:20.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:20.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:21.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:21.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:21.743 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:15:21.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:15:21.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:21.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:22.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:23.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:23.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:23.318 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:15:23.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:15:23.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:23.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:24.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:25.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:25.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:25.819 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:15:25.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:15:25.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:25.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:26.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:27.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:27.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:27.900 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:15:27.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:15:27.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:27.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:28.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:29.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:29.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:29.761 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:15:29.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:15:29.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:29.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:30.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:31.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:31.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:31.567 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:15:31.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:15:31.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:31.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:32.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:33.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:33.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:33.354 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:15:33.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:15:33.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:33.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:34.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:35.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:35.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:35.246 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:15:35.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:15:35.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:35.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:15:35.763 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3484108317,v1:127.0.0.1:6811/3484108317] [v2:127.0.0.1:6812/3484108317,v1:127.0.0.1:6813/3484108317] exists,up 0bd636e1-b8bf-4cf5-b78e-735ee32cdf7f 2024-04-25T08:15:35.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:15:35.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:15:35.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:15:35.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2024-04-25T08:15:35.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:15:35.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:15:35.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:15:35.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:15:35.768 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:15:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:15:35.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:15:35.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc0f3217-1f59-49fe-8b7a-bd06a63f7ff3 2024-04-25T08:15:35.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dc0f3217-1f59-49fe-8b7a-bd06a63f7ff3' 2024-04-25T08:15:35.771 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 dc0f3217-1f59-49fe-8b7a-bd06a63f7ff3 2024-04-25T08:15:35.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:15:35.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnESpm1oIRLxAAJx4ZkawMQv3bVfhGTlPbAw== 2024-04-25T08:15:35.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnESpm1oIRLxAAJx4ZkawMQv3bVfhGTlPbAw=="}' 2024-04-25T08:15:35.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc0f3217-1f59-49fe-8b7a-bd06a63f7ff3 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:15:36.342 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:15:36.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:15:36.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnESpm1oIRLxAAJx4ZkawMQv3bVfhGTlPbAw== --osd-uuid dc0f3217-1f59-49fe-8b7a-bd06a63f7ff3 2024-04-25T08:15:36.395 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:36.395+0000 7f21ce5f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:15:36.397 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:36.395+0000 7f21ce5f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:15:36.398 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:36.399+0000 7f21ce5f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:15:36.399 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:36.399+0000 7f21ce5f97c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:15:36.399 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:36.399+0000 7f21ce5f97c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:15:36.399 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:36.399+0000 7f21ce5f97c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:15:36.459 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:36.459+0000 7f21ce5f97c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:15:45.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:15:45.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:15:45.081 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:15:45.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:15:45.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:15:45.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:15:45.671 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:15:45.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:15:45.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:15:45.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:15:45.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:15:45.702 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:45.698+0000 7fcfbd3537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:15:45.704 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:45.702+0000 7fcfbd3537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:15:45.705 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:45.702+0000 7fcfbd3537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:15:46.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:15:46.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:15:46.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:15:46.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:15:46.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:15:46.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:46.064 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:15:46.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:15:46.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:46.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:46.257 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:46.254+0000 7fcfbd3537c0 -1 Falling back to public interface 2024-04-25T08:15:46.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:47.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:47.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:47.467 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:15:47.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:15:47.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:47.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:47.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:48.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:48.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:48.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:15:48.873 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:15:48.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:48.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:49.105 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:15:49.102+0000 7fcfbd3537c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:15:49.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:50.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:50.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:50.414 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:15:50.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:15:50.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:50.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:51.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:52.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:52.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:52.088 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:15:52.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:15:52.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:52.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:52.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:53.656 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:15:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:15:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:54.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:55.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:55.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:55.228 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:15:55.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:15:55.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:55.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:55.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:56.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:56.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:56.756 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:15:56.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:15:56.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:56.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:57.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:58.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:58.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:58.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:15:58.183 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:15:58.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:58.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:15:58.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:15:59.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:15:59.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:15:59.794 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:15:59.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:15:59.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:15:59.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:00.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:01.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:01.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:01.514 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:16:01.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:16:01.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:01.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:02.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:03.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:03.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:03.068 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:16:03.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:16:03.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:03.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:03.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:04.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:04.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:04.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:16:04.654 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:16:04.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:04.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:05.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:06.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:06.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:06.115 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:16:06.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:16:06.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:06.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:06.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:07.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:07.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:07.666 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:16:07.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:16:07.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:07.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:08.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:09.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:09.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:09.504 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:16:09.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:16:09.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:09.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:10.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:11.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:11.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:11.261 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:16:11.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:16:11.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:11.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:12.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:13.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:13.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:13.207 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:16:13.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:16:13.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:13.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:14.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:15.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:15.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:15.258 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:16:15.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:16:15.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:15.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:16.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:17.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:17.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:17.363 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:16:17.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:16:17.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:17.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:18.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:19.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:19.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:19.640 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:16:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:16:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:19.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:20.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:21.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:21.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:21.849 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:16:21.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:16:21.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:21.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:23.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:23.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:23.359 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:16:23.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:16:23.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:23.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:23.795 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/748976057,v1:127.0.0.1:6819/748976057] [v2:127.0.0.1:6820/748976057,v1:127.0.0.1:6821/748976057] exists,up dc0f3217-1f59-49fe-8b7a-bd06a63f7ff3 2024-04-25T08:16:23.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:16:23.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:16:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:16:23.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:16:23.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:16:23.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:16:23.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:16:23.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:16:23.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:16:23.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:16:23.799 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:16:23.799 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:23.799 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:16:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:16:23.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:16:23.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1798a42-3e55-4f73-a371-6cccc7b8a362 2024-04-25T08:16:23.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d1798a42-3e55-4f73-a371-6cccc7b8a362' 2024-04-25T08:16:23.802 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 d1798a42-3e55-4f73-a371-6cccc7b8a362 2024-04-25T08:16:23.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:16:23.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXESpmA1AAMRAAYrqWmC/CllKowwk+VtHAYg== 2024-04-25T08:16:23.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXESpmA1AAMRAAYrqWmC/CllKowwk+VtHAYg=="}' 2024-04-25T08:16:23.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1798a42-3e55-4f73-a371-6cccc7b8a362 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:16:24.384 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:16:24.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:16:24.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXESpmA1AAMRAAYrqWmC/CllKowwk+VtHAYg== --osd-uuid d1798a42-3e55-4f73-a371-6cccc7b8a362 2024-04-25T08:16:24.430 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:24.429+0000 7f3a025eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:24.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:24.433+0000 7f3a025eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:24.434 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:24.433+0000 7f3a025eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:24.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:24.433+0000 7f3a025eb7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:16:24.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:24.433+0000 7f3a025eb7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:16:24.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:24.433+0000 7f3a025eb7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:16:24.501 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:24.501+0000 7f3a025eb7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:16:32.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:16:32.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:16:32.747 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T08:16:32.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:16:32.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:16:33.310 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T08:16:33.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:16:33.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:16:33.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:16:33.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:16:33.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:16:33.341 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:33.341+0000 7f83267ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:33.344 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:33.345+0000 7f83267ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:33.345 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:33.345+0000 7f83267ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:33.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:16:33.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:16:33.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:16:33.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:16:33.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:16:33.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:33.751 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:16:33.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:16:33.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:33.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:33.912 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:33.913+0000 7f83267ed7c0 -1 Falling back to public interface 2024-04-25T08:16:34.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:35.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:35.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:35.150 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:16:35.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:16:35.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:35.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:35.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:36.584 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:16:36.585+0000 7f83267ed7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:16:36.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:36.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:36.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:16:36.585 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:16:36.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:36.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:37.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:38.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:38.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:38.088 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:16:38.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:16:38.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:38.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:38.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:39.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:39.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:39.519 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:16:39.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:16:39.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:39.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:40.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:41.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:41.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:41.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:16:41.200 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:16:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:41.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:41.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:42.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:42.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:42.740 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:16:42.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:16:42.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:42.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:43.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:44.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:44.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:44.259 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:16:44.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:16:44.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:44.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:44.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:45.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:45.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:45.796 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:16:45.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:16:45.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:45.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:46.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:47.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:47.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:47.258 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:16:47.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:16:47.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:47.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:47.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:48.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:48.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:48.793 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:16:48.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:16:48.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:48.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:49.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:50.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:50.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:50.236 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:16:50.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:16:50.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:50.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:50.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:51.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:51.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:51.743 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:16:51.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:16:51.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:51.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:52.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:53.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:53.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:53.341 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:16:53.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:16:53.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:53.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:54.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:54.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:54.814 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:16:54.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:16:54.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:54.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:55.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:56.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:56.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:56.604 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:16:56.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:16:56.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:56.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:57.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:58.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:58.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:16:58.312 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:16:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:58.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:59.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:00.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:00.046 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:17:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:17:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:00.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:00.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:01.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:01.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:01.945 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:17:01.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:17:01.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:01.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:02.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:03.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:03.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:03.746 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:17:03.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:17:03.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:03.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:05.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:05.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:05.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:17:05.508 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:17:05.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:05.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:06.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:07.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:07.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:07.565 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:17:07.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:17:07.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:07.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:08.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:09.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:09.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:09.607 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:17:09.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:17:09.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:09.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:10.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:11.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:11.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:11.146 INFO:tasks.workunit.client.0.smithi088.stdout:23 2024-04-25T08:17:11.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-04-25T08:17:11.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:11.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:11.589 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3227024692,v1:127.0.0.1:6827/3227024692] [v2:127.0.0.1:6828/3227024692,v1:127.0.0.1:6829/3227024692] exists,up d1798a42-3e55-4f73-a371-6cccc7b8a362 2024-04-25T08:17:11.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:17:11.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:17:11.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:17:11.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T08:17:12.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2024-04-25T08:17:12.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T08:17:13.605 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T08:17:13.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:14.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2024-04-25T08:17:15.713 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 2 2024-04-25T08:17:15.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2024-04-25T08:17:15.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:17:15.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:17:15.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:17:15.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:17:15.739 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:17:15.739 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:17:15.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:17:15.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:17:15.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:17:15.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:17:15.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:17:15.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:17:15.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:17:15.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:17:15.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:17:16.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:17:16.284 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:17:16.284 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:17:16.284 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:17:16.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:17:16.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:16.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:17:16.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836515 2024-04-25T08:17:16.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836515 2024-04-25T08:17:16.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515' 2024-04-25T08:17:16.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:16.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:17:16.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-04-25T08:17:16.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-04-25T08:17:16.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672989' 2024-04-25T08:17:16.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:16.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:17:16.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509463 2024-04-25T08:17:16.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509463 2024-04-25T08:17:16.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672989 2-64424509463' 2024-04-25T08:17:16.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:16.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:17:16.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345933 2024-04-25T08:17:16.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345933 2024-04-25T08:17:16.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672989 2-64424509463 3-85899345933' 2024-04-25T08:17:16.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:16.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836515 2024-04-25T08:17:16.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:16.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:17:16.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836515 2024-04-25T08:17:16.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:16.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836515 2024-04-25T08:17:16.875 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836515 2024-04-25T08:17:16.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-04-25T08:17:16.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:17:17.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836515 -lt 21474836515 2024-04-25T08:17:17.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:17.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-04-25T08:17:17.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:17.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:17:17.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-04-25T08:17:17.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:17.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-04-25T08:17:17.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-04-25T08:17:17.316 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672989 2024-04-25T08:17:17.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:17:17.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-04-25T08:17:17.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:17.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:17.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509463 2024-04-25T08:17:17.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:17:17.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:17.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509463 2024-04-25T08:17:17.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509463 2024-04-25T08:17:17.720 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509463 2024-04-25T08:17:17.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-04-25T08:17:17.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:17:18.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509461 -lt 64424509463 2024-04-25T08:17:18.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:17:19.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:17:19.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:17:19.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509464 -lt 64424509463 2024-04-25T08:17:19.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:19.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345933 2024-04-25T08:17:19.593 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:19.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:17:19.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345933 2024-04-25T08:17:19.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:19.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345933 2024-04-25T08:17:19.596 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345933 2024-04-25T08:17:19.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-04-25T08:17:19.597 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:17:20.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345933 -lt 85899345933 2024-04-25T08:17:20.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:17:20.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:20.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:20.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:17:20.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:17:20.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:17:20.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:17:20.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:17:20.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:17:20.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:17:20.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:17:21.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:17:21.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:17:21.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:17:21.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:17:21.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:17:21.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:17:21.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:17:21.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2024-04-25T08:17:21.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:21.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2024-04-25T08:17:21.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:21.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2024-04-25T08:17:22.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:22.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2024-04-25T08:17:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:22.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2024-04-25T08:17:22.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:22.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2024-04-25T08:17:22.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:22.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2024-04-25T08:17:23.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:23.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2024-04-25T08:17:23.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:23.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2024-04-25T08:17:23.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:23.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2024-04-25T08:17:24.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:24.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2024-04-25T08:17:24.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:24.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2024-04-25T08:17:24.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:24.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2024-04-25T08:17:24.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:24.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2024-04-25T08:17:25.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:25.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2024-04-25T08:17:25.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:25.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2024-04-25T08:17:25.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:25.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2024-04-25T08:17:25.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:25.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2024-04-25T08:17:26.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:26.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2024-04-25T08:17:26.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:26.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2024-04-25T08:17:26.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:26.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2024-04-25T08:17:27.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:27.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2024-04-25T08:17:27.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:27.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2024-04-25T08:17:27.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:27.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2024-04-25T08:17:28.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:28.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2024-04-25T08:17:28.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:28.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2024-04-25T08:17:28.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:28.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2024-04-25T08:17:28.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:28.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2024-04-25T08:17:29.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:29.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2024-04-25T08:17:29.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:29.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2024-04-25T08:17:29.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:29.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2024-04-25T08:17:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:29.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2024-04-25T08:17:30.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:30.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2024-04-25T08:17:30.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:30.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2024-04-25T08:17:30.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:30.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2024-04-25T08:17:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:31.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2024-04-25T08:17:31.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:31.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2024-04-25T08:17:31.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:31.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2024-04-25T08:17:31.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:31.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2024-04-25T08:17:32.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:32.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2024-04-25T08:17:32.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:32.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2024-04-25T08:17:32.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:32.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2024-04-25T08:17:32.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:32.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2024-04-25T08:17:33.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:33.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2024-04-25T08:17:33.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:33.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2024-04-25T08:17:33.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:33.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2024-04-25T08:17:33.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:33.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2024-04-25T08:17:34.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:34.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2024-04-25T08:17:34.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:34.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2024-04-25T08:17:34.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:34.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2024-04-25T08:17:35.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:35.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2024-04-25T08:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:35.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2024-04-25T08:17:35.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:35.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2024-04-25T08:17:35.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:35.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2024-04-25T08:17:36.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:36.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2024-04-25T08:17:36.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:36.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2024-04-25T08:17:36.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:36.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2024-04-25T08:17:36.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:36.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2024-04-25T08:17:37.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:37.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2024-04-25T08:17:37.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:37.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2024-04-25T08:17:37.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:37.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2024-04-25T08:17:38.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:38.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2024-04-25T08:17:38.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:38.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2024-04-25T08:17:38.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:38.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2024-04-25T08:17:38.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:38.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2024-04-25T08:17:39.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:39.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2024-04-25T08:17:39.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:39.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2024-04-25T08:17:39.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:39.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2024-04-25T08:17:39.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:39.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2024-04-25T08:17:40.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:40.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2024-04-25T08:17:40.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:40.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2024-04-25T08:17:40.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:40.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2024-04-25T08:17:41.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:41.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2024-04-25T08:17:41.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:41.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2024-04-25T08:17:41.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:41.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2024-04-25T08:17:41.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:41.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2024-04-25T08:17:42.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:42.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2024-04-25T08:17:42.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:42.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2024-04-25T08:17:42.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:42.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2024-04-25T08:17:43.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:43.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2024-04-25T08:17:43.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:43.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2024-04-25T08:17:43.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:43.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2024-04-25T08:17:43.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:43.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2024-04-25T08:17:44.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:44.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2024-04-25T08:17:44.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:44.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2024-04-25T08:17:44.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:44.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2024-04-25T08:17:45.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:45.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2024-04-25T08:17:45.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:45.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2024-04-25T08:17:45.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:45.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2024-04-25T08:17:45.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:45.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2024-04-25T08:17:46.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:46.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2024-04-25T08:17:46.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:46.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2024-04-25T08:17:46.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:46.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2024-04-25T08:17:46.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:46.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2024-04-25T08:17:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:47.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2024-04-25T08:17:47.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:47.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2024-04-25T08:17:47.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:47.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2024-04-25T08:17:48.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:48.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2024-04-25T08:17:48.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:48.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2024-04-25T08:17:48.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:48.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2024-04-25T08:17:48.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:48.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2024-04-25T08:17:49.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:49.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2024-04-25T08:17:49.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:49.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2024-04-25T08:17:49.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:49.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2024-04-25T08:17:50.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:50.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2024-04-25T08:17:50.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:50.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2024-04-25T08:17:50.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:50.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2024-04-25T08:17:50.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:50.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2024-04-25T08:17:51.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:51.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2024-04-25T08:17:51.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:51.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2024-04-25T08:17:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2024-04-25T08:17:51.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:51.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2024-04-25T08:17:52.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:52.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2024-04-25T08:17:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:52.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2024-04-25T08:17:52.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:52.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2024-04-25T08:17:53.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:53.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2024-04-25T08:17:53.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:53.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2024-04-25T08:17:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2024-04-25T08:17:53.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:53.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2024-04-25T08:17:54.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:54.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2024-04-25T08:17:54.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:54.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2024-04-25T08:17:54.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:54.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2024-04-25T08:17:54.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:54.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2024-04-25T08:17:55.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:55.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2024-04-25T08:17:55.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:55.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2024-04-25T08:17:55.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:55.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2024-04-25T08:17:56.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:56.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2024-04-25T08:17:56.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:56.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2024-04-25T08:17:56.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:56.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2024-04-25T08:17:56.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:56.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2024-04-25T08:17:57.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:57.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2024-04-25T08:17:57.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:57.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2024-04-25T08:17:58.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:58.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2024-04-25T08:17:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2024-04-25T08:17:58.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:58.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2024-04-25T08:17:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:58.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2024-04-25T08:17:59.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:59.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2024-04-25T08:17:59.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:59.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2024-04-25T08:17:59.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:17:59.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2024-04-25T08:18:00.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:00.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2024-04-25T08:18:00.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:00.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2024-04-25T08:18:00.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:00.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2024-04-25T08:18:00.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:00.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2024-04-25T08:18:01.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:01.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2024-04-25T08:18:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2024-04-25T08:18:01.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:01.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2024-04-25T08:18:02.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:02.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2024-04-25T08:18:02.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:02.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2024-04-25T08:18:02.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:02.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2024-04-25T08:18:02.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:02.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2024-04-25T08:18:03.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:03.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2024-04-25T08:18:03.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:03.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2024-04-25T08:18:03.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:03.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2024-04-25T08:18:04.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:04.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2024-04-25T08:18:04.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:04.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2024-04-25T08:18:04.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:04.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2024-04-25T08:18:04.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:04.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2024-04-25T08:18:05.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:05.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2024-04-25T08:18:05.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:05.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2024-04-25T08:18:05.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:05.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2024-04-25T08:18:06.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:06.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2024-04-25T08:18:06.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:06.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2024-04-25T08:18:06.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:06.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2024-04-25T08:18:06.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:06.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2024-04-25T08:18:07.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:07.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2024-04-25T08:18:07.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:07.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2024-04-25T08:18:07.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:07.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2024-04-25T08:18:08.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:08.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2024-04-25T08:18:08.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:08.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2024-04-25T08:18:08.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:08.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2024-04-25T08:18:09.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:09.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2024-04-25T08:18:09.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:09.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2024-04-25T08:18:09.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:09.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2024-04-25T08:18:09.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:09.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2024-04-25T08:18:10.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:10.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2024-04-25T08:18:10.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:10.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2024-04-25T08:18:10.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:10.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2024-04-25T08:18:11.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:11.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2024-04-25T08:18:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2024-04-25T08:18:11.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:11.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2024-04-25T08:18:11.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:11.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2024-04-25T08:18:12.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:12.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2024-04-25T08:18:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2024-04-25T08:18:12.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:12.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2024-04-25T08:18:12.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:12.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2024-04-25T08:18:13.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:13.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2024-04-25T08:18:13.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:13.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2024-04-25T08:18:13.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:13.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2024-04-25T08:18:14.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:14.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2024-04-25T08:18:14.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:14.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2024-04-25T08:18:14.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:14.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2024-04-25T08:18:14.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:14.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2024-04-25T08:18:15.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:15.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2024-04-25T08:18:15.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:15.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2024-04-25T08:18:15.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:15.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2024-04-25T08:18:15.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:15.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2024-04-25T08:18:16.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:16.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2024-04-25T08:18:16.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:16.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2024-04-25T08:18:16.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:16.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2024-04-25T08:18:17.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:17.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2024-04-25T08:18:17.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:18:17.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2024-04-25T08:18:17.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2024-04-25T08:18:17.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:18:17.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:18:17.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:18:17.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:18:18.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2024-04-25T08:18:18.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2024-04-25T08:18:18.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:18:18.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:18:18.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:18:18.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:18:18.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2024-04-25T08:18:18.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2024-04-25T08:18:18.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:18:18.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:18:18.648 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:18:18.648 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:18:18.648 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:18:18.649 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:18:18.649 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:18:19.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:18:19.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:18:19.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:18:19.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2024-04-25T08:18:19.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2024-04-25T08:18:20.344 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is set 2024-04-25T08:18:20.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:18:20.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 293587 2024-04-25T08:18:20.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2024-04-25T08:18:21.114 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T08:18:21.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2024-04-25T08:18:22.075 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T08:18:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:18:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2024-04-25T08:18:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2024-04-25T08:18:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2024-04-25T08:18:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:18:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2024-04-25T08:18:22.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2024-04-25T08:18:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:18:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2024-04-25T08:18:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2024-04-25T08:18:22.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2024-04-25T08:18:22.793 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already out. 2024-04-25T08:18:22.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2024-04-25T08:18:22.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2024-04-25T08:18:23.588 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is unset 2024-04-25T08:18:23.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2024-04-25T08:18:23.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:18:23.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:18:23.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:18:23.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:18:24.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:18:24.192 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:18:24.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2024-04-25T08:18:26.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2024-04-25T08:18:26.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:18:26.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:18:26.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:18:26.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:18:26.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:18:26.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:18:26.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:18:26.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:18:26.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:18:26.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:18:26.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:18:26.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:18:26.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:18:26.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:18:26.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:18:26.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:18:26.782 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:18:26.782 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:18:26.782 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:18:26.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:18:26.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:26.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:18:27.025 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:18:27.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:18:27.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:18:27.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:18:27.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:27.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:18:27.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673005 2024-04-25T08:18:27.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673005 2024-04-25T08:18:27.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673005' 2024-04-25T08:18:27.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:27.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:18:27.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509479 2024-04-25T08:18:27.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509479 2024-04-25T08:18:27.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673005 2-64424509479' 2024-04-25T08:18:27.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:27.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:18:27.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345949 2024-04-25T08:18:27.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345949 2024-04-25T08:18:27.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673005 2-64424509479 3-85899345949' 2024-04-25T08:18:27.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:27.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673005 2024-04-25T08:18:27.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:27.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:18:27.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673005 2024-04-25T08:18:27.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:27.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673005 2024-04-25T08:18:27.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-04-25T08:18:27.499 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673005 2024-04-25T08:18:27.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:18:27.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673003 -lt 42949673005 2024-04-25T08:18:27.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:18:28.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:18:28.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:18:29.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673005 -lt 42949673005 2024-04-25T08:18:29.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:29.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509479 2024-04-25T08:18:29.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:29.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:18:29.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509479 2024-04-25T08:18:29.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:29.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509479 2024-04-25T08:18:29.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-04-25T08:18:29.573 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509479 2024-04-25T08:18:29.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:18:30.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509480 -lt 64424509479 2024-04-25T08:18:30.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:30.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:30.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345949 2024-04-25T08:18:30.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:18:30.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345949 2024-04-25T08:18:30.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:30.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345949 2024-04-25T08:18:30.015 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345949 2024-04-25T08:18:30.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345949' 2024-04-25T08:18:30.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:18:30.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345949 -lt 85899345949 2024-04-25T08:18:30.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:18:30.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:30.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:31.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:18:31.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:31.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:31.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:31.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:31.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:31.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:31.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:31.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:31.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:31.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:31.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:32.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:32.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:18:32.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:18:32.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:18:32.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:32.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:32.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:32.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:32.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:32.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:32.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:32.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:32.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:32.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:33.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:33.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:33.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:33.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:33.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:33.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:33.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:33.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:33.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:33.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:33.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:33.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:33.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:34.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:18:34.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:18:34.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:34.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:34.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:34.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:34.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:34.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:34.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:34.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:34.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:34.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:34.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:34.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:34.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:34.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:34.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:35.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:35.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:35.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:35.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:35.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:35.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:35.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:35.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:35.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:36.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:18:36.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:18:36.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:36.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:36.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:36.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:36.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:36.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:36.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:36.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:36.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:36.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:36.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:36.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:36.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:36.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:36.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:37.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:37.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:37.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:37.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:37.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:37.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:37.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:37.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:37.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:37.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:18:37.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:18:37.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:37.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:37.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:37.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:37.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:37.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:37.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:37.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:37.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:37.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:37.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:38.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:38.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:38.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:38.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:38.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:38.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:38.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:38.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:38.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:38.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:38.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:38.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:38.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:39.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:18:39.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:18:39.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:39.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:39.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:39.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:39.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:39.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:39.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:39.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:39.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:39.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:39.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:40.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:40.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:40.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:40.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:40.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:40.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:40.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:40.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:40.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:40.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:40.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:40.685 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:40.685 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:18:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:18:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:41.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:41.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:41.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:41.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:41.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:41.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:41.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:41.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:41.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:41.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:41.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:41.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:42.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:42.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:42.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:42.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:42.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:42.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:42.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:42.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:42.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:43.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:18:43.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:18:43.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:43.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:43.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:43.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:43.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:43.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:43.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:43.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:43.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:43.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:43.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:43.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:43.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:43.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:43.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:44.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:44.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:44.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:44.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:44.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:44.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:44.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:44.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:44.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:44.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:18:44.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:18:44.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:44.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:44.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:45.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:45.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:45.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:45.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:45.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:45.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:45.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:45.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:45.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:45.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:45.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:45.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:46.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:46.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:46.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:46.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:46.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:46.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:46.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:46.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:46.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:46.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:18:46.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:18:46.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:46.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:46.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:46.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:46.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:46.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:46.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:46.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:46.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:46.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:46.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:47.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:47.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:47.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:47.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:47.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:47.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:47.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:47.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:47.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:47.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:47.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:47.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:47.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:48.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-04-25T08:18:48.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-04-25T08:18:48.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:48.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:48.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:48.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:48.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:48.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:48.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:48.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:48.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:48.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:48.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:49.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:18:49.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:49.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:49.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:49.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:18:49.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:18:49.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:18:49.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:18:49.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:18:49.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:18:49.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:18:49.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:18:49.644 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:18:50.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:18:50.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:18:50.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:18:50.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:18:50.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:18:50.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:18:50.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:50.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:50.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:50.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:50.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:50.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:50.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:50.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:18:50.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:50.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:50.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:51.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:18:51.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:18:51.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:18:51.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:18:51.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-04-25T08:18:51.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-04-25T08:18:51.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:18:51.266 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:18:51.266 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:18:51.266 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:18:51.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2024-04-25T08:18:51.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:18:51.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:18:51.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:18:51.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-04-25T08:18:51.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:18:51.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:18:51.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:18:51.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:18:51.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:18:51.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:18:51.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:18:51.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:18:51.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:18:51.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:18:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:18:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:18:51.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:18:51.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:18:51.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:18:51.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:18:51.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:18:51.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:18:51.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:18:51.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:18:51.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:18:51.310 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:18:51.310 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-04-25T08:18:51.310 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:18:51.310 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:18:51.311 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:18:51.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2024-04-25T08:18:51.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2024-04-25T08:18:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:18:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:18:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:18:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:18:51.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:18:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:18:51.319 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:18:51.319 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2024-04-25T08:18:51.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:18:51.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:18:51.884 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T08:18:51.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2024-04-25T08:18:51.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:18:51.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:18:51.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:18:51.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:18:51.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:18:57.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:18:57.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:18:57.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:18:57.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:18:57.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:18:57.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:18:57.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:18:57.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:18:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:18:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:18:57.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:18:57.232 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:18:57.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:18:57.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:18:57.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:18:57.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:18:57.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:18:57.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:18:57.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:18:57.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:18:57.235 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:18:57.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:18:57.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:18:57.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:18:57.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:18:57.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:57.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:18:57.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:18:57.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:18:57.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:18:57.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:18:57.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:18:57.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:18:57.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:18:57.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:18:57.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:18:57.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:18:57.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:18:57.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:18:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:18:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:18:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:18:57.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:18:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:18:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:18:57.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:18:57.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:18:57.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:57.285 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:18:57.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:18:57.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:18:57.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:18:57.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:18:57.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:18:57.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:57.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:18:57.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274961 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:18:57.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:18:57.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:18:57.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:18:57.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:18:57.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:18:57.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:18:57.813 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:18:57.813 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:57.813 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:18:57.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:18:57.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:18:58.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:18:58.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:18:58.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:18:58.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:18:58.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:18:58.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:18:58.040 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:18:58.040 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:18:58.040 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:18:58.040 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:58.041 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:58.041 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:18:58.041 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:18:58.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:18:58.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get fsid 2024-04-25T08:18:58.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:18:58.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:18:58.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:18:58.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:18:58.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:18:58.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:18:58.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:18:58.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:18:58.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:58.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:58.154 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:18:58.154 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:18:58.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:18:58.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get mon_host 2024-04-25T08:18:58.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2024-04-25T08:18:58.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:18:58.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:18:58.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:18:58.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:18:58.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:18:58.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:18:59.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:18:59.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:18:59.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:18:59.186 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:18:59.186 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:59.186 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:18:59.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:18:59.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:18:59.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:18:59.221 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2024-04-25T08:18:59.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:18:59.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:59.226 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:18:59.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:18:59.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1f89746e-621d-421c-8913-ee3d548378da 2024-04-25T08:18:59.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1f89746e-621d-421c-8913-ee3d548378da' 2024-04-25T08:18:59.228 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 1f89746e-621d-421c-8913-ee3d548378da 2024-04-25T08:18:59.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:18:59.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzESpmVOOnDhAAPaIpdo/h9hN5Cn+sxvktwg== 2024-04-25T08:18:59.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzESpmVOOnDhAAPaIpdo/h9hN5Cn+sxvktwg=="}' 2024-04-25T08:18:59.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1f89746e-621d-421c-8913-ee3d548378da -i td/osd-recovery-stats/0/new.json 2024-04-25T08:18:59.601 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:18:59.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:18:59.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzESpmVOOnDhAAPaIpdo/h9hN5Cn+sxvktwg== --osd-uuid 1f89746e-621d-421c-8913-ee3d548378da 2024-04-25T08:18:59.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:18:59.650+0000 7fd22ac1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:18:59.677 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:18:59.678+0000 7fd22ac1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:18:59.679 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:18:59.678+0000 7fd22ac1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:18:59.679 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:18:59.682+0000 7fd22ac1c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:18:59.679 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:18:59.682+0000 7fd22ac1c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:18:59.679 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:18:59.682+0000 7fd22ac1c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:18:59.717 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:18:59.718+0000 7fd22ac1c7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:19:07.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:19:07.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:19:07.523 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:19:07.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:19:07.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:19:08.109 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:19:08.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:19:08.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:19:08.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:19:08.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:19:08.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:19:08.141 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:08.141+0000 7f80bc1957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:08.143 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:08.145+0000 7f80bc1957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:08.144 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:08.145+0000 7f80bc1957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:08.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:19:08.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:19:08.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:19:08.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:19:08.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:19:08.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:08.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:19:08.528 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:19:08.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:08.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:08.692 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:08.693+0000 7f80bc1957c0 -1 Falling back to public interface 2024-04-25T08:19:08.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:09.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:09.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:09.925 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:19:09.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:19:09.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:09.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:10.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:11.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:11.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:11.338 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:19:11.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:19:11.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:11.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:11.502 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:11.501+0000 7f80bc1957c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:19:11.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:12.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:12.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:12.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:19:12.775 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:19:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:13.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:14.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:14.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:14.238 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:19:14.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:19:14.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:14.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:14.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:15.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:15.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:15.839 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:19:15.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:19:15.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:15.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:16.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:17.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:17.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:17.337 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:19:17.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:19:17.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:17.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:17.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:18.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:18.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:18.950 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:19:18.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:19:18.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:18.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:19.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:20.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:20.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:20.554 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:19:20.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:19:20.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:20.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:21.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:22.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:22.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:22.035 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:19:22.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:19:22.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:22.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:22.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:23.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:23.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:23.597 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:19:23.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:19:23.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:23.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:24.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:25.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:25.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:25.157 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:19:25.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:19:25.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:25.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:25.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:26.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:26.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:26.793 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:19:26.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:19:26.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:26.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:27.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:28.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:28.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:28.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:19:28.572 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:19:28.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:28.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:29.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:30.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:30.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:30.025 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:19:30.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:19:30.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:30.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:30.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:31.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:31.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:31.635 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:19:31.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:19:31.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:31.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:32.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:33.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:33.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:33.318 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:19:33.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:19:33.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:33.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:34.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:35.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:35.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:35.250 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:19:35.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:19:35.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:35.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:37.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:37.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:37.323 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:19:37.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:19:37.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:37.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:38.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:39.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:39.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:39.367 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:19:39.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:19:39.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:39.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:40.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:41.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:41.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:41.381 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:19:41.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:19:41.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:41.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:42.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:43.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:43.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:43.181 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:19:43.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:19:43.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:43.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:19:43.800 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/290205732,v1:127.0.0.1:6803/290205732] [v2:127.0.0.1:6804/290205732,v1:127.0.0.1:6805/290205732] exists,up 1f89746e-621d-421c-8913-ee3d548378da 2024-04-25T08:19:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:19:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:19:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:19:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:19:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:19:43.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:19:43.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:19:43.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:19:43.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:19:43.804 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:19:43.804 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:19:43.804 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:19:43.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f31e403-c11f-437d-8ee5-b91d2b01f4a9 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9f31e403-c11f-437d-8ee5-b91d2b01f4a9' 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 9f31e403-c11f-437d-8ee5-b91d2b01f4a9 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:19:43.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfEipmZNFsMRAAbq+7G20XOzxxk7wyob/+XA== 2024-04-25T08:19:43.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfEipmZNFsMRAAbq+7G20XOzxxk7wyob/+XA=="}' 2024-04-25T08:19:43.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f31e403-c11f-437d-8ee5-b91d2b01f4a9 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:19:44.351 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:19:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:19:44.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfEipmZNFsMRAAbq+7G20XOzxxk7wyob/+XA== --osd-uuid 9f31e403-c11f-437d-8ee5-b91d2b01f4a9 2024-04-25T08:19:44.402 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:44.401+0000 7f07545dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:44.404 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:44.405+0000 7f07545dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:44.405+0000 7f07545dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:44.405+0000 7f07545dd7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:19:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:44.405+0000 7f07545dd7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:19:44.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:44.405+0000 7f07545dd7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:19:44.468 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:44.469+0000 7f07545dd7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:19:52.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:19:52.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:19:52.678 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:19:52.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:19:52.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:19:53.260 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:19:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:19:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:19:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:19:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:19:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:19:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:53.288+0000 7f29da7967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:53.293 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:53.292+0000 7f29da7967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:53.294 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:53.292+0000 7f29da7967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:19:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:19:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:19:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:19:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:19:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:19:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:19:53.675 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:19:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:19:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:53.832+0000 7f29da7967c0 -1 Falling back to public interface 2024-04-25T08:19:54.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:55.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:55.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:55.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:19:55.073 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:19:55.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:55.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:19:55.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:56.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:56.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:56.478 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:19:56.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:19:56.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:56.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:19:56.806 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:19:56.804+0000 7f29da7967c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:19:56.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:57.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:57.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:57.916 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:19:57.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:19:57.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:57.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:19:58.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:19:59.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:19:59.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:19:59.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:19:59.585 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:19:59.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:19:59.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:00.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:01.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:01.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:01.456 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:20:01.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:20:01.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:01.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:02.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:03.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:03.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:03.246 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:20:03.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:20:03.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:03.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:03.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:04.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:04.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:04.676 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:20:04.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:20:04.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:04.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:05.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:06.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:06.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:06.380 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:20:06.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:20:06.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:06.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:06.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:07.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:07.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:07.968 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:20:07.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:20:07.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:07.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:08.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:09.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:09.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:09.521 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:20:09.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:20:09.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:09.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:10.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:11.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:11.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:11.181 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:20:11.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:20:11.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:11.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:11.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:12.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:12.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:12.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:20:12.692 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:20:12.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:12.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:13.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:14.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:14.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:14.217 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:20:14.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:20:14.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:15.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:16.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:16.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:16.006 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:20:16.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:20:16.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:16.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:16.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:17.635 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:20:17.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:20:17.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:17.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:18.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:19.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:19.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:19.964 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:20:19.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:20:19.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:19.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:20.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:21.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:21.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:21.712 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:20:21.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:20:21.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:21.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:22.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:23.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:23.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:23.721 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:20:23.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:20:23.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:23.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:24.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:25.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:25.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:25.781 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:20:25.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:20:25.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:25.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:27.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:28.093 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:20:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:20:28.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:28.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:29.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:30.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:30.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:30.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:20:30.399 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:20:30.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:30.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:20:30.834 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1447386706,v1:127.0.0.1:6811/1447386706] [v2:127.0.0.1:6812/1447386706,v1:127.0.0.1:6813/1447386706] exists,up 9f31e403-c11f-437d-8ee5-b91d2b01f4a9 2024-04-25T08:20:30.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:20:30.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:20:30.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:20:30.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:20:30.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:20:30.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:20:30.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:20:30.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5958440-f998-43ea-bd25-165e1cef25ce 2024-04-25T08:20:30.841 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 c5958440-f998-43ea-bd25-165e1cef25ce 2024-04-25T08:20:30.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c5958440-f998-43ea-bd25-165e1cef25ce' 2024-04-25T08:20:30.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:20:30.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOEipmStVrMxAAtVXt6/1WMiGo49Y+V3cCUg== 2024-04-25T08:20:30.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOEipmStVrMxAAtVXt6/1WMiGo49Y+V3cCUg=="}' 2024-04-25T08:20:30.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5958440-f998-43ea-bd25-165e1cef25ce -i td/osd-recovery-stats/2/new.json 2024-04-25T08:20:31.423 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:20:31.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:20:31.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOEipmStVrMxAAtVXt6/1WMiGo49Y+V3cCUg== --osd-uuid c5958440-f998-43ea-bd25-165e1cef25ce 2024-04-25T08:20:31.474 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:31.472+0000 7f149ba5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:31.476 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:31.476+0000 7f149ba5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:31.478 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:31.476+0000 7f149ba5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:31.478 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:31.476+0000 7f149ba5e7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:20:31.478 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:31.476+0000 7f149ba5e7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:20:31.478 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:31.476+0000 7f149ba5e7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:20:31.557 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:31.556+0000 7f149ba5e7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:20:40.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:20:40.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:20:40.102 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:20:40.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:20:40.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:20:40.646 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:20:40.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:20:40.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:20:40.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:20:40.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:20:40.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:20:40.678 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:40.675+0000 7f157060b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:40.706 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:40.703+0000 7f157060b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:40.708 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:40.707+0000 7f157060b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:20:41.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:20:41.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:20:41.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:20:41.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:20:41.082 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:20:41.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:20:41.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:41.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:20:41.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:41.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:41.266 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:41.263+0000 7f157060b7c0 -1 Falling back to public interface 2024-04-25T08:20:41.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:42.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:42.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:42.483 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:20:42.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:20:42.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:42.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:42.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:43.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:43.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:43.886 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:20:43.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:20:43.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:43.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:44.148 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:20:44.147+0000 7f157060b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:20:44.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:45.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:45.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:45.297 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:20:45.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:20:45.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:45.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:45.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:46.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:46.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:46.804 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:20:46.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:20:46.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:46.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:47.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:48.655 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:20:48.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:20:48.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:48.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:49.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:50.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:50.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:50.117 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:20:50.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:20:50.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:50.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:50.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:51.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:51.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:51.661 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:20:51.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:20:51.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:51.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:52.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:53.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:53.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:53.082 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:20:53.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:20:53.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:53.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:53.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:54.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:54.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:54.711 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:20:54.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:20:54.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:54.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:55.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:56.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:56.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:56.359 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:20:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:20:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:56.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:56.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:57.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:57.927 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:20:57.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:20:57.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:57.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:20:58.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:20:59.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:20:59.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:20:59.531 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:20:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:20:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:20:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:00.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:01.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:01.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:01.059 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:21:01.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:21:01.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:01.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:01.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:02.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:02.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:02.715 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:21:02.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:21:02.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:02.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:03.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:21:04.276 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:21:04.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:04.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:05.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:06.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:06.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:06.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:21:06.181 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:21:06.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:06.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:07.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:08.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:08.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:08.417 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:21:08.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:21:08.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:08.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:09.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:10.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:10.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:10.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:21:10.305 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:21:10.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:10.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:11.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:12.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:21:12.302 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:21:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:13.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:14.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:14.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:14.636 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:21:14.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:21:14.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:14.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:15.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:16.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:16.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:16.342 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:21:16.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:21:16.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:16.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:16.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:17.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:17.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:17.929 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:21:17.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:21:17.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:17.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:21:18.351 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1486678545,v1:127.0.0.1:6819/1486678545] [v2:127.0.0.1:6820/1486678545,v1:127.0.0.1:6821/1486678545] exists,up c5958440-f998-43ea-bd25-165e1cef25ce 2024-04-25T08:21:18.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:21:18.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:21:18.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:21:18.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:21:18.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2024-04-25T08:21:18.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:21:18.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:21:18.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:21:18.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:21:18.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:21:18.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:21:18.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:21:18.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:21:18.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:21:18.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9efac83e-c1e5-4308-a783-7e034357b447 2024-04-25T08:21:18.361 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 9efac83e-c1e5-4308-a783-7e034357b447 2024-04-25T08:21:18.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9efac83e-c1e5-4308-a783-7e034357b447' 2024-04-25T08:21:18.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:21:18.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+EipmoXxgFhAABCGETDaVSQPY6vqmbj+G/g== 2024-04-25T08:21:18.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+EipmoXxgFhAABCGETDaVSQPY6vqmbj+G/g=="}' 2024-04-25T08:21:18.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9efac83e-c1e5-4308-a783-7e034357b447 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:21:19.081 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:21:19.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:21:19.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+EipmoXxgFhAABCGETDaVSQPY6vqmbj+G/g== --osd-uuid 9efac83e-c1e5-4308-a783-7e034357b447 2024-04-25T08:21:19.135 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:19.130+0000 7f91cc1387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:19.138 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:19.134+0000 7f91cc1387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:19.139 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:19.134+0000 7f91cc1387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:19.140 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:19.138+0000 7f91cc1387c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:21:19.140 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:19.138+0000 7f91cc1387c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:21:19.140 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:19.138+0000 7f91cc1387c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:21:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:19.218+0000 7f91cc1387c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:21:27.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:21:27.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:21:27.258 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T08:21:27.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:21:27.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:21:27.831 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T08:21:27.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:21:27.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:21:27.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:21:27.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:21:27.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:21:27.862 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:27.858+0000 7fcfe1bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:27.865 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:27.862+0000 7fcfe1bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:27.866 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:27.862+0000 7fcfe1bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:21:28.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:21:28.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:21:28.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:21:28.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:21:28.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:21:28.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:28.245 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:21:28.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:21:28.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:28.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:28.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:28.418+0000 7fcfe1bfc7c0 -1 Falling back to public interface 2024-04-25T08:21:28.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:29.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:29.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:29.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:21:29.673 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:21:29.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:29.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:30.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:31.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:31.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:31.099 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:21:31.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:21:31.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:31.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:31.249 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:21:31.246+0000 7fcfe1bfc7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:21:31.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:32.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:32.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:32.565 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:21:32.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:21:32.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:32.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:33.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:34.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:34.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:34.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:21:34.353 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:21:34.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:34.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:34.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:35.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:35.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:35.900 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:21:35.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:21:35.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:35.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:36.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:37.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:37.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:37.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:21:37.461 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:21:37.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:37.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:37.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:38.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:38.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:38.914 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:21:38.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:21:38.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:38.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:39.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:40.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:40.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:40.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:21:40.618 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:21:40.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:40.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:41.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:42.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:42.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:42.140 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:21:42.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:21:42.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:42.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:42.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:43.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:43.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:43.726 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:21:43.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:21:43.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:43.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:44.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:45.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:45.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:45.171 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:21:45.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:21:45.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:45.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:45.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:46.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:46.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:46.795 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:21:46.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:21:46.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:46.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:47.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:48.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:48.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:48.501 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:21:48.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:21:48.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:48.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:49.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:50.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:50.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:50.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:21:50.112 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:21:50.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:50.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:50.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:51.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:51.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:51.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:21:51.624 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:21:51.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:51.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:52.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:53.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:53.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:53.424 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:21:53.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:21:53.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:53.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:54.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:55.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:55.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:55.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:21:55.173 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:21:55.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:55.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:55.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:56.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:56.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:56.983 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:21:56.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:21:56.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:56.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:57.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:21:58.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:21:58.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:21:58.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:21:58.952 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:21:58.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:21:58.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:21:59.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:00.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:00.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:00.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:22:00.918 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:22:00.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:00.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:22:01.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:02.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:02.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:02.754 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:22:02.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:22:02.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:02.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:22:03.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:04.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:04.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:04.723 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:22:04.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:22:04.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:04.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:22:05.267 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3053513508,v1:127.0.0.1:6827/3053513508] [v2:127.0.0.1:6828/3053513508,v1:127.0.0.1:6829/3053513508] exists,up 9efac83e-c1e5-4308-a783-7e034357b447 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:22:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:22:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:22:05.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:22:05.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:22:05.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:22:05.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:22:05.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:22:05.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1bd92e71-51a8-44a3-98ab-1e0f9a4cde92 2024-04-25T08:22:05.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 1bd92e71-51a8-44a3-98ab-1e0f9a4cde92' 2024-04-25T08:22:05.275 INFO:tasks.workunit.client.0.smithi088.stdout:add osd4 1bd92e71-51a8-44a3-98ab-1e0f9a4cde92 2024-04-25T08:22:05.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:22:05.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtEipmLrbaERAAIsZog+LLyeq9JnNtgJqySg== 2024-04-25T08:22:05.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtEipmLrbaERAAIsZog+LLyeq9JnNtgJqySg=="}' 2024-04-25T08:22:05.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1bd92e71-51a8-44a3-98ab-1e0f9a4cde92 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:22:06.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:22:06.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtEipmLrbaERAAIsZog+LLyeq9JnNtgJqySg== --osd-uuid 1bd92e71-51a8-44a3-98ab-1e0f9a4cde92 2024-04-25T08:22:06.037 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:06.037+0000 7f98a37757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:06.039 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:06.037+0000 7f98a37757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:06.040 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:06.041+0000 7f98a37757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:06.041 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:06.041+0000 7f98a37757c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:22:06.041 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:06.041+0000 7f98a37757c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:22:06.041 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:06.041+0000 7f98a37757c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:22:06.159 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:06.157+0000 7f98a37757c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:22:14.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:22:14.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:22:14.450 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd4 key to auth repository 2024-04-25T08:22:14.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:22:14.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:22:15.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:22:15.037 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.4 2024-04-25T08:22:15.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:22:15.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:22:15.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:22:15.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:22:15.068 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:15.069+0000 7f94dd60f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:15.070 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:15.069+0000 7f94dd60f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:15.071 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:15.073+0000 7f94dd60f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:22:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:22:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:22:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:22:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:22:15.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:15.436 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:22:15.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:22:15.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:15.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:15.622 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:15.621+0000 7f94dd60f7c0 -1 Falling back to public interface 2024-04-25T08:22:15.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:16.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:16.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:16.839 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:22:16.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:22:16.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:16.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:17.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:18.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:18.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:18.264 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:22:18.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:22:18.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:18.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:18.605 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:18.605+0000 7f94dd60f7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:22:18.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:19.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:19.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:19.717 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:22:19.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:22:19.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:19.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:20.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:21.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:21.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:21.574 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:22:21.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:22:21.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:21.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:22.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:23.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:23.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:23.277 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:22:23.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:22:23.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:23.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:23.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:24.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:24.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:24.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:22:24.866 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:22:24.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:24.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:25.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:26.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:26.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:26.307 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:22:26.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:22:26.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:26.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:26.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:27.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:27.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:27.716 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:22:27.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:22:27.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:27.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:28.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:29.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:29.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:29.225 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:22:29.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:22:29.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:29.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:29.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:30.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:30.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:30.833 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:22:30.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:22:30.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:30.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:31.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:32.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:32.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:32.333 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:22:32.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:22:32.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:32.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:32.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:33.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:33.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:33.892 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:22:33.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:22:33.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:33.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:34.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:35.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:35.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:35.376 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:22:35.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:22:35.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:35.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:36.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:37.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:37.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:37.097 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:22:37.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:22:37.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:37.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:37.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:38.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:38.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:38.747 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:22:38.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:22:38.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:38.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:39.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:40.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:40.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:40.481 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:22:40.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:22:40.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:40.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:41.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:42.320 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:22:42.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:22:42.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:42.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:43.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:44.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:44.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:44.345 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:22:44.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:22:44.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:44.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:45.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:46.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:46.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:46.395 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:22:46.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:22:46.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:46.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:47.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:48.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:48.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:48.951 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:22:48.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:22:48.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:48.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:49.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:50.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:50.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:50.772 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:22:50.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:22:50.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:50.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:51.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:22:52.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:22:52.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:22:52.301 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:22:52.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:22:52.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:22:52.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:22:52.780 INFO:tasks.workunit.client.0.smithi088.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/626431395,v1:127.0.0.1:6835/626431395] [v2:127.0.0.1:6836/626431395,v1:127.0.0.1:6837/626431395] exists,up 1bd92e71-51a8-44a3-98ab-1e0f9a4cde92 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:22:52.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:22:52.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:22:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:22:52.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:22:52.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89f7a777-8d56-4e37-9028-e5e91bbe38ae 2024-04-25T08:22:52.787 INFO:tasks.workunit.client.0.smithi088.stdout:add osd5 89f7a777-8d56-4e37-9028-e5e91bbe38ae 2024-04-25T08:22:52.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 89f7a777-8d56-4e37-9028-e5e91bbe38ae' 2024-04-25T08:22:52.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:22:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcEipmAHovMBAACgL/sMGE+k/2tP++2Pue8w== 2024-04-25T08:22:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcEipmAHovMBAACgL/sMGE+k/2tP++2Pue8w=="}' 2024-04-25T08:22:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89f7a777-8d56-4e37-9028-e5e91bbe38ae -i td/osd-recovery-stats/5/new.json 2024-04-25T08:22:53.352 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:22:53.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:22:53.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcEipmAHovMBAACgL/sMGE+k/2tP++2Pue8w== --osd-uuid 89f7a777-8d56-4e37-9028-e5e91bbe38ae 2024-04-25T08:22:53.403 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:53.400+0000 7f5e6c6257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:53.484+0000 7f5e6c6257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:53.484+0000 7f5e6c6257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:22:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:53.484+0000 7f5e6c6257c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:22:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:53.484+0000 7f5e6c6257c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:22:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:53.484+0000 7f5e6c6257c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:22:53.613 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:22:53.612+0000 7f5e6c6257c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:23:01.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:23:01.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:23:01.670 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd5 key to auth repository 2024-04-25T08:23:01.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:23:01.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:23:02.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:23:02.238 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.5 2024-04-25T08:23:02.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:23:02.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:23:02.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:23:02.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:23:02.270 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:23:02.268+0000 7ff14ea6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:02.280 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:23:02.280+0000 7ff14ea6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:02.281 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:23:02.280+0000 7ff14ea6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:23:02.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:23:02.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:23:02.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:23:02.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:23:02.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:23:02.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:02.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:23:02.671 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:23:02.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:02.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:02.842 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:23:02.840+0000 7ff14ea6b7c0 -1 Falling back to public interface 2024-04-25T08:23:03.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:04.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:04.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:04.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:23:04.121 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:23:04.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:04.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:04.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:05.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:05.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:05.569 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:23:05.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:23:05.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:05.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:05.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:23:05.624+0000 7ff14ea6b7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:23:06.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:07.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:07.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:07.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:23:07.084 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:23:07.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:07.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:07.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:08.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:08.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:08.497 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:23:08.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:23:08.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:08.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:08.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:09.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:09.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:09.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:23:09.987 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:23:09.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:09.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:10.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:11.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:11.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:11.441 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:23:11.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:23:11.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:11.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:11.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:12.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:12.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:12.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:23:12.999 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:23:12.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:12.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:13.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:14.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:14.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:14.824 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:23:14.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:23:14.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:14.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:15.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:16.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:16.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:16.284 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:23:16.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:23:16.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:16.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:17.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:18.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:18.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:18.058 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:23:18.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:23:18.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:18.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:18.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:19.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:19.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:19.730 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:23:19.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:23:19.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:19.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:20.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:21.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:21.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:21.258 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:23:21.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:23:21.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:21.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:21.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:23:22.822 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:23:22.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:22.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:23.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:24.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:24.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:24.288 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:23:24.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:23:24.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:24.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:24.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:25.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:25.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:25.785 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:23:25.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:23:25.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:25.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:26.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:27.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:27.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:27.567 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:23:27.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:23:27.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:27.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:28.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:29.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:29.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:29.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:23:29.675 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:23:29.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:29.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:30.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:31.755 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:23:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:23:31.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:31.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:32.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:33.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:33.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:33.766 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:23:33.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:23:33.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:33.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:34.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:35.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:35.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:35.686 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:23:35.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:23:35.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:35.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:36.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:37.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:37.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:37.589 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:23:37.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:23:37.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:37.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:38.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:23:39.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:23:39.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:23:39.325 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:23:39.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:23:39.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:23:39.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:23:39.786 INFO:tasks.workunit.client.0.smithi088.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/15531112,v1:127.0.0.1:6843/15531112] [v2:127.0.0.1:6844/15531112,v1:127.0.0.1:6845/15531112] exists,up 89f7a777-8d56-4e37-9028-e5e91bbe38ae 2024-04-25T08:23:39.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:23:39.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:23:39.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:23:39.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2024-04-25T08:23:39.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:23:40.404 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T08:23:40.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:23:41.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2024-04-25T08:23:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 3 2024-04-25T08:23:42.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2024-04-25T08:23:43.800 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 1 2024-04-25T08:23:43.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2024-04-25T08:23:43.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:23:43.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:23:43.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:23:43.823 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:23:43.824 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:23:43.824 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:23:43.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:23:43.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:23:43.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:23:43.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:23:43.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:23:43.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:23:43.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:23:43.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:23:43.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:23:44.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:23:44.635 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:23:44.635 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:23:44.635 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:23:44.635 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:23:44.635 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:23:44.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:23:44.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:23:44.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:23:44.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836537 2024-04-25T08:23:44.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836537 2024-04-25T08:23:44.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537' 2024-04-25T08:23:44.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:23:44.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:23:44.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673004 2024-04-25T08:23:44.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673004 2024-04-25T08:23:44.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673004' 2024-04-25T08:23:44.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:23:44.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:23:45.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509472 2024-04-25T08:23:45.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509472 2024-04-25T08:23:45.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673004 2-64424509472' 2024-04-25T08:23:45.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:23:45.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:23:45.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345945 2024-04-25T08:23:45.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345945 2024-04-25T08:23:45.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673004 2-64424509472 3-85899345945' 2024-04-25T08:23:45.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:23:45.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:23:45.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182415 2024-04-25T08:23:45.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182415 2024-04-25T08:23:45.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673004 2-64424509472 3-85899345945 4-107374182415' 2024-04-25T08:23:45.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:23:45.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:23:45.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018890 2024-04-25T08:23:45.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018890 2024-04-25T08:23:45.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673004 2-64424509472 3-85899345945 4-107374182415 5-128849018890' 2024-04-25T08:23:45.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:23:45.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836537 2024-04-25T08:23:45.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:23:45.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:23:45.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836537 2024-04-25T08:23:45.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:23:45.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836537 2024-04-25T08:23:45.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-04-25T08:23:45.543 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836537 2024-04-25T08:23:45.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:23:45.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836534 -lt 21474836537 2024-04-25T08:23:45.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:23:46.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:23:46.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:23:47.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836537 -lt 21474836537 2024-04-25T08:23:47.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:23:47.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673004 2024-04-25T08:23:47.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:23:47.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:23:47.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673004 2024-04-25T08:23:47.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:23:47.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673004 2024-04-25T08:23:47.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-04-25T08:23:47.372 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673004 2024-04-25T08:23:47.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:23:47.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673005 -lt 42949673004 2024-04-25T08:23:47.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:23:47.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509472 2024-04-25T08:23:47.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:23:47.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:23:47.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509472 2024-04-25T08:23:47.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:23:47.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509472 2024-04-25T08:23:47.808 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509472 2024-04-25T08:23:47.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-04-25T08:23:47.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:23:48.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509472 -lt 64424509472 2024-04-25T08:23:48.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:23:48.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345945 2024-04-25T08:23:48.197 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:23:48.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:23:48.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:23:48.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345945 2024-04-25T08:23:48.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345945 2024-04-25T08:23:48.200 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345945 2024-04-25T08:23:48.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345945' 2024-04-25T08:23:48.201 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:23:48.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345946 -lt 85899345945 2024-04-25T08:23:48.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:23:48.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182415 2024-04-25T08:23:48.640 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:23:48.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:23:48.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:23:48.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182415 2024-04-25T08:23:48.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182415 2024-04-25T08:23:48.644 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182415 2024-04-25T08:23:48.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-04-25T08:23:48.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:23:49.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182416 -lt 107374182415 2024-04-25T08:23:49.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:23:49.061 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018890 2024-04-25T08:23:49.062 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:23:49.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:23:49.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018890 2024-04-25T08:23:49.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:23:49.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018890 2024-04-25T08:23:49.066 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 128849018890 2024-04-25T08:23:49.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2024-04-25T08:23:49.066 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:23:49.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018890 -lt 128849018890 2024-04-25T08:23:49.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:23:49.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:49.524 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:50.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:23:50.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:50.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:50.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:50.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:50.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:50.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:50.049 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:50.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:23:50.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:50.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:50.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:51.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:23:51.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:23:51.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:23:51.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2024-04-25T08:23:51.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:23:51.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:23:51.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:23:51.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:23:51.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:23:51.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2024-04-25T08:23:51.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2024-04-25T08:23:51.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:23:51.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:23:51.725 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:23:51.725 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:23:51.725 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:23:51.725 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:23:51.725 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:23:52.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:23:52.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:23:52.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:23:52.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2024-04-25T08:23:52.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2024-04-25T08:23:53.426 INFO:tasks.workunit.client.0.smithi088.stderr:noout is set 2024-04-25T08:23:53.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2024-04-25T08:23:54.355 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is set 2024-04-25T08:23:54.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:23:54.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 307292 2024-04-25T08:23:54.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2024-04-25T08:23:55.304 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T08:23:55.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2024-04-25T08:23:55.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2024-04-25T08:23:55.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2024-04-25T08:23:55.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:55.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2024-04-25T08:23:55.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:55.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2024-04-25T08:23:56.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:56.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2024-04-25T08:23:56.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:56.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2024-04-25T08:23:56.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:56.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2024-04-25T08:23:56.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:56.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2024-04-25T08:23:56.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:56.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2024-04-25T08:23:57.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:57.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2024-04-25T08:23:57.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:57.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2024-04-25T08:23:57.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:57.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2024-04-25T08:23:57.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:57.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2024-04-25T08:23:57.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:57.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2024-04-25T08:23:57.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:57.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2024-04-25T08:23:57.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:57.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2024-04-25T08:23:58.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:58.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2024-04-25T08:23:58.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:58.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2024-04-25T08:23:58.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:58.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2024-04-25T08:23:58.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:58.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2024-04-25T08:23:58.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:58.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2024-04-25T08:23:58.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:58.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2024-04-25T08:23:59.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:59.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2024-04-25T08:23:59.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:59.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2024-04-25T08:23:59.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:59.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2024-04-25T08:23:59.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:59.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2024-04-25T08:23:59.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:59.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2024-04-25T08:23:59.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:23:59.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2024-04-25T08:24:00.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:00.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2024-04-25T08:24:00.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:00.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2024-04-25T08:24:00.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:00.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2024-04-25T08:24:00.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:00.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2024-04-25T08:24:00.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:00.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2024-04-25T08:24:00.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:00.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2024-04-25T08:24:01.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:01.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2024-04-25T08:24:01.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:01.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2024-04-25T08:24:01.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:01.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2024-04-25T08:24:01.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:01.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2024-04-25T08:24:01.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:01.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2024-04-25T08:24:01.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:01.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2024-04-25T08:24:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2024-04-25T08:24:02.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:02.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2024-04-25T08:24:02.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:02.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2024-04-25T08:24:02.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:02.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2024-04-25T08:24:02.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:02.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2024-04-25T08:24:02.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:02.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2024-04-25T08:24:02.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:02.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2024-04-25T08:24:03.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:03.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2024-04-25T08:24:03.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:03.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2024-04-25T08:24:03.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:03.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2024-04-25T08:24:03.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:03.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2024-04-25T08:24:03.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:03.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2024-04-25T08:24:03.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:03.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2024-04-25T08:24:04.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:04.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2024-04-25T08:24:04.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:04.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2024-04-25T08:24:04.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:04.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2024-04-25T08:24:04.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:04.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2024-04-25T08:24:04.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:04.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2024-04-25T08:24:04.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:04.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2024-04-25T08:24:04.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:04.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2024-04-25T08:24:05.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:05.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2024-04-25T08:24:05.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:05.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2024-04-25T08:24:05.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:05.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2024-04-25T08:24:05.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:05.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2024-04-25T08:24:05.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:05.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2024-04-25T08:24:05.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:05.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2024-04-25T08:24:06.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:06.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2024-04-25T08:24:06.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:06.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2024-04-25T08:24:06.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:06.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2024-04-25T08:24:06.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:06.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2024-04-25T08:24:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:06.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2024-04-25T08:24:06.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:06.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2024-04-25T08:24:07.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:07.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2024-04-25T08:24:07.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:07.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2024-04-25T08:24:07.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:07.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2024-04-25T08:24:07.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:07.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2024-04-25T08:24:07.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:07.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2024-04-25T08:24:07.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:07.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2024-04-25T08:24:08.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:08.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2024-04-25T08:24:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:08.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2024-04-25T08:24:08.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:08.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2024-04-25T08:24:08.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:08.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2024-04-25T08:24:08.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:08.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2024-04-25T08:24:08.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:08.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2024-04-25T08:24:09.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:09.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2024-04-25T08:24:09.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:09.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2024-04-25T08:24:09.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:09.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2024-04-25T08:24:09.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:09.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2024-04-25T08:24:09.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:09.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2024-04-25T08:24:09.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:09.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2024-04-25T08:24:09.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:09.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2024-04-25T08:24:10.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:10.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2024-04-25T08:24:10.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:10.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2024-04-25T08:24:10.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:10.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2024-04-25T08:24:10.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:10.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2024-04-25T08:24:10.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:10.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2024-04-25T08:24:10.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:10.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2024-04-25T08:24:11.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:11.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2024-04-25T08:24:11.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:11.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2024-04-25T08:24:11.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:11.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2024-04-25T08:24:11.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:24:11.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2024-04-25T08:24:11.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2024-04-25T08:24:11.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 309130 2024-04-25T08:24:11.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2024-04-25T08:24:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T08:24:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2024-04-25T08:24:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-04-25T08:24:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:24:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T08:24:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:24:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:24:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:24:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:24:12.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:24:12.351 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:24:12.351 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:12.351 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:24:12.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:24:12.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:24:12.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:24:12.352 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:24:12.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T08:24:12.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:24:12.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/0/whoami 2024-04-25T08:24:12.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T08:24:12.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:24:12.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:24:12.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:24:12.389 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:12.387+0000 7f048f7197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:12.444 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:12.443+0000 7f048f7197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:12.446 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:12.443+0000 7f048f7197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:12.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T08:24:12.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:24:12.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:24:12.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:24:12.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:24:12.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:12.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:24:12.925 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:24:12.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:24:12.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:13.018 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:13.015+0000 7f048f7197c0 -1 Falling back to public interface 2024-04-25T08:24:13.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:14.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:14.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:14.348 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:24:14.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:24:14.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:14.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:24:14.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:15.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:15.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:15.771 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:24:15.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:24:15.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:15.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:24:16.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:16.683 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:16.679+0000 7f048f7197c0 -1 osd.0 39 log_to_monitors true 2024-04-25T08:24:17.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:17.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:17.235 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:24:17.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:24:17.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:17.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:24:17.859 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 48 up_thru 48 down_at 43 last_clean_interval [5,39) [v2:127.0.0.1:6802/68542659,v1:127.0.0.1:6803/68542659] [v2:127.0.0.1:6804/68542659,v1:127.0.0.1:6805/68542659] exists,up 1f89746e-621d-421c-8913-ee3d548378da 2024-04-25T08:24:17.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:24:17.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:24:17.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:24:17.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2024-04-25T08:24:20.863 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2024-04-25T08:24:20.865 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2024-04-25T08:24:20.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:20.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2024-04-25T08:24:21.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:21.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2024-04-25T08:24:21.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:21.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2024-04-25T08:24:21.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:21.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2024-04-25T08:24:21.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:21.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2024-04-25T08:24:21.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:21.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2024-04-25T08:24:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:21.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2024-04-25T08:24:21.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:21.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2024-04-25T08:24:21.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:21.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2024-04-25T08:24:22.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:22.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2024-04-25T08:24:22.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:22.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2024-04-25T08:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:22.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2024-04-25T08:24:22.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2024-04-25T08:24:22.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:22.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2024-04-25T08:24:22.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:22.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2024-04-25T08:24:22.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:22.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2024-04-25T08:24:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2024-04-25T08:24:23.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:23.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2024-04-25T08:24:23.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:23.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2024-04-25T08:24:23.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:23.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2024-04-25T08:24:23.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:23.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2024-04-25T08:24:23.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:23.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2024-04-25T08:24:23.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:23.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2024-04-25T08:24:24.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:24.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2024-04-25T08:24:24.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:24.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2024-04-25T08:24:24.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:24.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2024-04-25T08:24:24.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:24.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2024-04-25T08:24:24.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:24.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2024-04-25T08:24:24.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:24.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2024-04-25T08:24:24.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:24.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2024-04-25T08:24:25.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:25.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2024-04-25T08:24:25.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:25.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2024-04-25T08:24:25.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:25.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2024-04-25T08:24:25.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:25.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2024-04-25T08:24:25.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:25.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2024-04-25T08:24:25.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:25.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2024-04-25T08:24:25.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:25.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2024-04-25T08:24:26.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:26.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2024-04-25T08:24:26.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:26.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2024-04-25T08:24:26.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:26.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2024-04-25T08:24:26.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:26.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2024-04-25T08:24:26.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:26.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2024-04-25T08:24:26.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:26.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2024-04-25T08:24:26.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:26.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2024-04-25T08:24:27.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:27.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2024-04-25T08:24:27.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:27.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2024-04-25T08:24:27.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:27.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2024-04-25T08:24:27.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:27.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2024-04-25T08:24:27.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:27.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2024-04-25T08:24:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2024-04-25T08:24:27.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:27.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2024-04-25T08:24:27.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:27.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2024-04-25T08:24:28.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:28.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2024-04-25T08:24:28.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:28.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2024-04-25T08:24:28.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:28.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2024-04-25T08:24:28.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:28.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2024-04-25T08:24:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:28.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2024-04-25T08:24:28.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:28.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2024-04-25T08:24:28.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:28.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2024-04-25T08:24:29.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:29.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2024-04-25T08:24:29.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:29.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2024-04-25T08:24:29.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:29.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2024-04-25T08:24:29.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:29.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2024-04-25T08:24:29.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:29.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2024-04-25T08:24:29.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:29.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2024-04-25T08:24:29.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:29.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2024-04-25T08:24:30.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:30.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2024-04-25T08:24:30.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:30.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2024-04-25T08:24:30.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:30.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2024-04-25T08:24:30.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:30.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2024-04-25T08:24:30.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:30.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2024-04-25T08:24:30.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:30.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2024-04-25T08:24:30.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:30.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2024-04-25T08:24:31.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:31.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2024-04-25T08:24:31.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:31.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2024-04-25T08:24:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:31.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2024-04-25T08:24:31.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:31.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2024-04-25T08:24:31.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:31.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2024-04-25T08:24:31.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:31.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2024-04-25T08:24:31.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:31.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2024-04-25T08:24:32.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:32.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2024-04-25T08:24:32.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:32.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2024-04-25T08:24:32.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:32.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2024-04-25T08:24:32.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:32.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2024-04-25T08:24:32.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:32.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2024-04-25T08:24:32.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:32.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2024-04-25T08:24:32.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:32.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2024-04-25T08:24:33.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:33.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2024-04-25T08:24:33.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:33.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2024-04-25T08:24:33.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:33.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2024-04-25T08:24:33.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:33.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2024-04-25T08:24:33.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:33.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2024-04-25T08:24:33.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:33.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2024-04-25T08:24:33.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:33.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2024-04-25T08:24:34.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:34.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2024-04-25T08:24:34.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:34.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2024-04-25T08:24:34.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:34.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2024-04-25T08:24:34.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:34.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2024-04-25T08:24:34.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:34.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2024-04-25T08:24:34.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:24:34.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2024-04-25T08:24:34.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2024-04-25T08:24:34.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:24:34.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:24:34.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:24:34.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:24:35.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2024-04-25T08:24:35.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2024-04-25T08:24:35.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:24:35.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj200 2024-04-25T08:24:35.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj200 2024-04-25T08:24:35.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:24:35.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj200 2024-04-25T08:24:35.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj200 2024-04-25T08:24:35.434 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:24:35.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=0 2024-04-25T08:24:35.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj200 2024-04-25T08:24:35.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2024-04-25T08:24:36.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2024-04-25T08:24:36.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2024-04-25T08:24:37.115 INFO:tasks.workunit.client.0.smithi088.stderr:noout is unset 2024-04-25T08:24:37.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2024-04-25T08:24:37.767 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already out. osd.2 is already out. 2024-04-25T08:24:37.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:24:37.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:24:37.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:24:37.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:24:37.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:24:37.786 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:24:37.786 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:37.786 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:24:37.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:24:37.788 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:24:37.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T08:24:37.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:24:37.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/1/whoami 2024-04-25T08:24:37.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T08:24:37.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:24:37.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:24:37.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:24:37.822 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:37.818+0000 7fe58c0d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:37.825 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:37.822+0000 7fe58c0d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:37.826 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:37.822+0000 7fe58c0d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:38.361 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:38.358+0000 7fe58c0d27c0 -1 Falling back to public interface 2024-04-25T08:24:38.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T08:24:38.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:24:38.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:24:38.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:24:38.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:24:38.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:38.445 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:24:38.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:24:38.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:38.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:39.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:40.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:40.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:40.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:24:40.155 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:24:40.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:40.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:40.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:41.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:41.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:41.624 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:24:41.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:24:41.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:41.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:42.013 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:24:42.010+0000 7fe58c0d27c0 -1 osd.1 44 log_to_monitors true 2024-04-25T08:24:42.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:24:43.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:24:43.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:24:43.104 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:24:43.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:24:43.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:24:43.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:24:43.623 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up out weight 0 up_from 56 up_thru 43 down_at 45 last_clean_interval [10,44) [v2:127.0.0.1:6810/765162152,v1:127.0.0.1:6811/765162152] [v2:127.0.0.1:6812/765162152,v1:127.0.0.1:6813/765162152] exists,up 9f31e403-c11f-437d-8ee5-b91d2b01f4a9 2024-04-25T08:24:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:24:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:24:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:24:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2024-04-25T08:24:46.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2024-04-25T08:24:47.656 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 4 2024-04-25T08:24:47.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2024-04-25T08:24:50.170 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is unset 2024-04-25T08:24:50.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:24:50.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:24:50.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:24:50.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:24:50.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:24:50.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T08:24:50.874 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:24:50.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2024-04-25T08:24:52.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2024-04-25T08:24:52.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:24:52.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:24:52.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:24:52.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:24:52.892 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:24:52.892 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:24:52.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:24:52.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:24:52.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:24:53.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:24:53.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:24:53.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:24:53.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:24:53.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:24:53.016 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:24:53.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:24:53.454 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:24:53.454 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:24:53.454 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:24:53.454 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:24:53.454 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:24:53.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:24:53.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:24:53.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:24:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=206158430218 2024-04-25T08:24:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 206158430218 2024-04-25T08:24:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430218' 2024-04-25T08:24:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:24:53.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:24:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=240518168581 2024-04-25T08:24:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 240518168581 2024-04-25T08:24:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430218 1-240518168581' 2024-04-25T08:24:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:24:53.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:24:54.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509488 2024-04-25T08:24:54.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509488 2024-04-25T08:24:54.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430218 1-240518168581 2-64424509488' 2024-04-25T08:24:54.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:24:54.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:24:54.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345961 2024-04-25T08:24:54.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345961 2024-04-25T08:24:54.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430218 1-240518168581 2-64424509488 3-85899345961' 2024-04-25T08:24:54.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:24:54.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:24:54.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182431 2024-04-25T08:24:54.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182431 2024-04-25T08:24:54.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430218 1-240518168581 2-64424509488 3-85899345961 4-107374182431' 2024-04-25T08:24:54.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:24:54.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:24:54.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018906 2024-04-25T08:24:54.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018906 2024-04-25T08:24:54.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-206158430218 1-240518168581 2-64424509488 3-85899345961 4-107374182431 5-128849018906' 2024-04-25T08:24:54.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:24:54.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-206158430218 2024-04-25T08:24:54.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:24:54.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:24:54.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-206158430218 2024-04-25T08:24:54.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:24:54.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=206158430218 2024-04-25T08:24:54.542 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 206158430218 2024-04-25T08:24:54.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 206158430218' 2024-04-25T08:24:54.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:24:54.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 206158430218 -lt 206158430218 2024-04-25T08:24:54.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:24:54.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-240518168581 2024-04-25T08:24:54.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:24:54.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:24:54.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-240518168581 2024-04-25T08:24:54.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:24:54.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=240518168581 2024-04-25T08:24:54.966 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 240518168581 2024-04-25T08:24:54.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 240518168581' 2024-04-25T08:24:54.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:24:55.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 240518168581 -lt 240518168581 2024-04-25T08:24:55.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:24:55.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509488 2024-04-25T08:24:55.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:24:55.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:24:55.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509488 2024-04-25T08:24:55.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:24:55.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509488 2024-04-25T08:24:55.549 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509488 2024-04-25T08:24:55.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509488' 2024-04-25T08:24:55.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:24:56.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509486 -lt 64424509488 2024-04-25T08:24:56.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:24:57.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:24:57.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:24:57.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509488 -lt 64424509488 2024-04-25T08:24:57.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:24:57.463 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345961 2024-04-25T08:24:57.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:24:57.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:24:57.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345961 2024-04-25T08:24:57.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:24:57.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345961 2024-04-25T08:24:57.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345961' 2024-04-25T08:24:57.468 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345961 2024-04-25T08:24:57.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:24:58.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345961 -lt 85899345961 2024-04-25T08:24:58.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:24:58.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182431 2024-04-25T08:24:58.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:24:58.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:24:58.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182431 2024-04-25T08:24:58.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:24:58.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182431 2024-04-25T08:24:58.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2024-04-25T08:24:58.104 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182431 2024-04-25T08:24:58.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:24:58.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182432 -lt 107374182431 2024-04-25T08:24:58.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:24:58.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018906 2024-04-25T08:24:58.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:24:58.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:24:58.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018906 2024-04-25T08:24:58.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:24:58.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018906 2024-04-25T08:24:58.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018906' 2024-04-25T08:24:58.673 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 128849018906 2024-04-25T08:24:58.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:24:59.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018906 -lt 128849018906 2024-04-25T08:24:59.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:24:59.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:59.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:00.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:25:00.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:00.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:00.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:00.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:00.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:00.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:00.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:00.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:00.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:00.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:00.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:25:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:25:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:25:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:01.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:01.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:01.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:01.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:01.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:01.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:01.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:01.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:01.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:01.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:01.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:01.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:01.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:02.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:02.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:02.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:02.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:02.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:02.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:02.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:02.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:02.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-04-25T08:25:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-04-25T08:25:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:02.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:02.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:02.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:02.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:02.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:02.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:02.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:03.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:03.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:03.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:03.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:04.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:04.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:04.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:04.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:04.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:04.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:04.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:04.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:04.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:04.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:25:04.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:25:04.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:04.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:04.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:04.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:04.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:04.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:04.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:04.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:04.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:04.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:04.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:05.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:05.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:05.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:05.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:05.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:05.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:05.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:05.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:05.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:05.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:05.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:05.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:05.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:06.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-04-25T08:25:06.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-04-25T08:25:06.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:06.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:06.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:06.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:06.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:06.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:06.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:06.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:06.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:06.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:06.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:07.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:07.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:07.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:07.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:07.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:07.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:07.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:07.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:07.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:07.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:07.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:07.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:07.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:08.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:25:08.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:25:08.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:08.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:08.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:08.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:08.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:08.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:08.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:08.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:08.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:08.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:08.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:09.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:09.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:09.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:09.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:10.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:10.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:10.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:10.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:10.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:10.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:10.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:10.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:10.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:10.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:25:10.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:25:10.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:10.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:10.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:10.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:10.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:10.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:10.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:10.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:10.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:10.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:10.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:11.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:11.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:11.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:11.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:11.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:11.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:11.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:11.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:11.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:11.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:11.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:11.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:11.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:12.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:25:12.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:25:12.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:12.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:12.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:12.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:12.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:12.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:12.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:12.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:12.777 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:13.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:13.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:13.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:13.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:13.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:13.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:13.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:13.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:13.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:13.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:13.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:13.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:13.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:14.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:25:14.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:25:14.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:14.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:14.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:14.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:14.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:14.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:14.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:14.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:14.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:14.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:14.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:15.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:15.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:15.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:15.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:15.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:15.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:15.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:15.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:15.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:15.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:15.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:15.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:15.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:25:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:25:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:16.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:16.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:16.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:16.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:16.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:16.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:16.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:16.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:16.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:16.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:16.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:16.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:17.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:17.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:17.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:18.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:25:18.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:25:18.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:18.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:18.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:18.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:18.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:18.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:18.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:18.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:18.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:18.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:18.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:18.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:18.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:18.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:18.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:19.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:19.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:19.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:19.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:19.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:19.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:19.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:19.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:19.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:20.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:25:20.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:25:20.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:20.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:20.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:20.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:20.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:20.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:20.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:20.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:20.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:20.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:20.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:20.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:20.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:20.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:20.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:21.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:21.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:21.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:21.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:21.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:21.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:21.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:21.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:21.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:25:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:25:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:22.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:22.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:22.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:22.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:22.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:22.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:22.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:22.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:22.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:22.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:22.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:22.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:22.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:23.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:23.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:23.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:23.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:23.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:23.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:23.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:23.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:23.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:24.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:25:24.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:25:24.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:24.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:24.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:24.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:24.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:24.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:24.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:24.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:24.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:24.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:24.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:24.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:24.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:24.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:24.634 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:25.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:25.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:26.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:25:26.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:25:26.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:26.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:26.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:26.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:26.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:26.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:26.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:26.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:26.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:26.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:26.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:26.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:26.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:26.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:26.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:27.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:27.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:27.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:27.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:27.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:27.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:27.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:27.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:27.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:28.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:25:28.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:25:28.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:28.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:28.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:28.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:28.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:28.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:28.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:28.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:28.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:28.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:28.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:28.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:28.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:28.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:28.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:29.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:29.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:29.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:29.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:29.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:29.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:29.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:29.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:29.361 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:25:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:25:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:29.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:30.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:30.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:30.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:30.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:30.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:30.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:30.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:30.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:30.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:30.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:30.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:30.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:31.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:31.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:31.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:31.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:31.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:31.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:31.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:31.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:31.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:32.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:25:32.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:25:32.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:32.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:32.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:32.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:32.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:32.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:32.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:32.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:32.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:32.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:32.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:32.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:32.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:32.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:32.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:33.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:33.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:33.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:33.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:34.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:25:34.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:25:34.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:34.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:34.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:34.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:34.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:34.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:34.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:34.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:34.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:34.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:34.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:34.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:34.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:34.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:34.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:35.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:35.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:35.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:35.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:35.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:35.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:35.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:35.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:35.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:36.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:25:36.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:25:36.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:36.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:36.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:36.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:36.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:36.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:36.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:36.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:36.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:36.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:36.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:36.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:25:36.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:36.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:36.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:37.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:25:37.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:25:37.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:25:37.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:25:37.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:25:37.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:25:37.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:25:37.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:25:37.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:25:37.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:25:37.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:25:37.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:25:37.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:25:37.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:25:37.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:25:37.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:25:37.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:25:37.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:25:37.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:25:37.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:25:37.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:25:37.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:25:38.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:25:38.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:25:38.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:25:38.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:25:38.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:25:38.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:25:38.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:25:38.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:25:38.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:25:38.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:25:38.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:25:38.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:25:39.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2024-04-25T08:25:39.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2024-04-25T08:25:39.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:25:39.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:25:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:25:39.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:25:39.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2024-04-25T08:25:39.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:25:39.270 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:25:39.270 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.270 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:25:39.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:25:39.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:25:39.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:25:39.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2024-04-25T08:25:39.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2024-04-25T08:25:39.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2024-04-25T08:25:39.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:25:39.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2024-04-25T08:25:39.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:25:39.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:25:39.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:25:39.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:25:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:25:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:25:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:25:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:25:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:25:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:25:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:25:39.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:25:39.289 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.289 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:25:39.289 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:25:39.290 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:25:39.290 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:25:39.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2024-04-25T08:25:39.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2024-04-25T08:25:39.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2024-04-25T08:25:39.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:25:39.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2024-04-25T08:25:39.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:25:39.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:25:39.295 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.295 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:25:39.295 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:25:39.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:25:39.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:25:39.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-04-25T08:25:39.307 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-04-25T08:25:39.308 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:25:39.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2024-04-25T08:25:39.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2024-04-25T08:25:39.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2024-04-25T08:25:39.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:25:39.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2024-04-25T08:25:39.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:25:39.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:25:39.313 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:25:39.313 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-04-25T08:25:39.313 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:25:39.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-04-25T08:25:39.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-04-25T08:25:39.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:25:39.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:25:39.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:25:39.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:25:39.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2024-04-25T08:25:39.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:25:39.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:25:39.914 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T08:25:39.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2024-04-25T08:25:39.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:25:39.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:25:39.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:25:39.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:25:39.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:25:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:25:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:25:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:25:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:25:41.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:25:41.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:25:41.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:25:41.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:25:41.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:25:41.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:25:41.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:25:41.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:25:41.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:25:41.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:25:41.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:25:41.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:25:41.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:25:41.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:25:41.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:25:41.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:25:41.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:25:41.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:25:41.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:25:41.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:25:41.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:25:41.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:41.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:25:41.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:25:41.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:25:41.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:25:41.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:25:41.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:25:41.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:25:41.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:25:41.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:25:41.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:25:41.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:25:41.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:25:41.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:25:41.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:25:41.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:25:41.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:25:41.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:25:41.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:25:41.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:25:41.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:41.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:25:41.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:25:41.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:25:41.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:25:41.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:25:41.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:25:41.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:41.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:25:41.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274961 2024-04-25T08:25:41.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:25:41.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:25:41.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:25:41.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:25:41.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:25:41.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:25:41.851 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:25:41.851 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:41.851 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:25:41.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:25:41.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:25:42.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:25:42.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:25:42.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:25:42.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:25:42.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:25:42.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:25:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:25:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:25:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:25:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:25:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:25:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:25:42.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:25:42.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get fsid 2024-04-25T08:25:42.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:25:42.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:25:42.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:25:42.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:25:42.263 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:25:42.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:25:42.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get mon_host 2024-04-25T08:25:42.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-04-25T08:25:42.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:25:42.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:25:42.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:25:42.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:25:42.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:25:42.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:25:44.123 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:25:44.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:25:44.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:25:44.124 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:25:44.124 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:44.124 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:25:44.124 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:25:44.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:25:44.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:25:44.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-04-25T08:25:44.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:25:44.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:25:44.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:25:44.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:25:44.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:25:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:25:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:25:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:25:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:25:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:25:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:25:44.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:25:44.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:25:44.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:25:44.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:25:44.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:25:44.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:25:44.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:25:44.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:44.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:25:44.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:25:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:25:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:25:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:25:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:25:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:25:44.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:25:44.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:25:44.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:25:44.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:25:44.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:25:44.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:25:44.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:25:44.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:25:44.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:25:44.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd5789cf-393a-47ec-9585-bb1f4d8eb1d0 2024-04-25T08:25:44.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dd5789cf-393a-47ec-9585-bb1f4d8eb1d0' 2024-04-25T08:25:44.212 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 dd5789cf-393a-47ec-9585-bb1f4d8eb1d0 2024-04-25T08:25:44.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:25:44.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIEypm/YFsDRAAlyzm+X9DhfNcYB9FKlZZWg== 2024-04-25T08:25:44.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIEypm/YFsDRAAlyzm+X9DhfNcYB9FKlZZWg=="}' 2024-04-25T08:25:44.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd5789cf-393a-47ec-9585-bb1f4d8eb1d0 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:25:44.573 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:25:44.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:25:44.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIEypm/YFsDRAAlyzm+X9DhfNcYB9FKlZZWg== --osd-uuid dd5789cf-393a-47ec-9585-bb1f4d8eb1d0 2024-04-25T08:25:44.627 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:44.625+0000 7fd1ef9927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:44.649 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:44.649+0000 7fd1ef9927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:44.651 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:44.649+0000 7fd1ef9927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:44.651 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:44.649+0000 7fd1ef9927c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:25:44.651 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:44.649+0000 7fd1ef9927c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:25:44.652 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:44.649+0000 7fd1ef9927c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:25:44.690 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:44.689+0000 7fd1ef9927c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:25:52.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:25:52.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:25:52.606 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:25:52.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:25:52.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:25:53.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:25:53.190 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:25:53.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:25:53.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:25:53.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:25:53.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:25:53.222 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:53.221+0000 7f6b8a60f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:53.224 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:53.225+0000 7f6b8a60f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:53.225 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:53.225+0000 7f6b8a60f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:53.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:25:53.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:25:53.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:25:53.596 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:25:53.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:25:53.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:25:53.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:53.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:25:53.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:53.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:53.786 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:53.785+0000 7f6b8a60f7c0 -1 Falling back to public interface 2024-04-25T08:25:53.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:54.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:54.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:54.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:25:54.998 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:25:54.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:54.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:55.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:56.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:56.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:56.409 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:25:56.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:25:56.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:56.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:56.711 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:25:56.708+0000 7f6b8a60f7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:25:56.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:57.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:57.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:57.828 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:25:57.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:25:57.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:57.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:58.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:59.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:59.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:59.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:25:59.311 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:25:59.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:59.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:59.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:00.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:00.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:00.773 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:26:00.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:26:00.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:00.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:01.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:02.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:02.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:02.290 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:26:02.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:26:02.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:02.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:02.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:03.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:03.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:03.934 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:26:03.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:26:03.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:03.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:04.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:05.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:05.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:05.408 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:26:05.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:26:05.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:05.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:05.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:06.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:06.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:06.971 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:26:06.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:26:06.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:06.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:08.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:08.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:08.408 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:26:08.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:26:08.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:08.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:09.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:10.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:10.094 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:26:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:26:10.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:10.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:10.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:11.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:11.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:11.561 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:26:11.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:26:11.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:11.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:12.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:13.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:13.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:13.108 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:26:13.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:26:13.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:13.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:13.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:14.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:14.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:14.773 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:26:14.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:26:14.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:14.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:15.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:16.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:16.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:16.255 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:26:16.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:26:16.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:16.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:17.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:18.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:18.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:18.070 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:26:18.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:26:18.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:18.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:19.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:20.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:20.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:20.041 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:26:20.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:26:20.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:20.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:21.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:22.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:22.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:22.088 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:26:22.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:26:22.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:22.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:23.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:24.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:24.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:24.223 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:26:24.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:26:24.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:24.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:24.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:25.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:25.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:25.928 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:26:25.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:26:25.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:25.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:26.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:27.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:26:27.696 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:26:27.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:27.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:28.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:29.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:29.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:29.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:26:29.480 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:26:29.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:29.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:26:29.898 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4094480428,v1:127.0.0.1:6803/4094480428] [v2:127.0.0.1:6804/4094480428,v1:127.0.0.1:6805/4094480428] exists,up dd5789cf-393a-47ec-9585-bb1f4d8eb1d0 2024-04-25T08:26:29.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:26:29.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:26:29.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:26:29.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-04-25T08:26:29.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:26:29.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:26:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:26:29.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:26:29.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:26:29.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:26:29.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:26:29.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:26:29.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:26:29.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:26:29.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:26:29.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b19a6f5d-140b-464a-9622-b95f07512e9c 2024-04-25T08:26:29.905 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 b19a6f5d-140b-464a-9622-b95f07512e9c 2024-04-25T08:26:29.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b19a6f5d-140b-464a-9622-b95f07512e9c' 2024-04-25T08:26:29.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:26:29.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1Eypmrjs+NxAA1mUg/RSh7kMDeYGDe5zm4g== 2024-04-25T08:26:29.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1Eypmrjs+NxAA1mUg/RSh7kMDeYGDe5zm4g=="}' 2024-04-25T08:26:29.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b19a6f5d-140b-464a-9622-b95f07512e9c -i td/osd-recovery-stats/1/new.json 2024-04-25T08:26:30.505 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:26:30.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:26:30.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1Eypmrjs+NxAA1mUg/RSh7kMDeYGDe5zm4g== --osd-uuid b19a6f5d-140b-464a-9622-b95f07512e9c 2024-04-25T08:26:30.553 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:30.552+0000 7f424e1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:30.554 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:30.552+0000 7f424e1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:30.556 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:30.556+0000 7f424e1cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:30.556 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:30.556+0000 7f424e1cd7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:26:30.556 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:30.556+0000 7f424e1cd7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:26:30.556 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:30.556+0000 7f424e1cd7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:26:30.630 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:30.628+0000 7f424e1cd7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:26:38.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:26:38.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:26:38.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:26:38.878 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:26:38.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:26:39.428 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:26:39.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:26:39.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:26:39.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:26:39.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:26:39.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:26:39.459 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:39.456+0000 7f42d843e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:39.502 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:39.500+0000 7f42d843e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:39.503 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:39.500+0000 7f42d843e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:26:39.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:26:39.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:26:39.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:26:39.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:26:39.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:26:39.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:39.824 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:26:39.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:26:39.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:39.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:40.062 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:40.060+0000 7f42d843e7c0 -1 Falling back to public interface 2024-04-25T08:26:40.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:26:41.241 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:26:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:41.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:41.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:42.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:42.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:42.656 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:26:42.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:26:42.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:42.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:42.882 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:26:42.879+0000 7f42d843e7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:26:43.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:44.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:44.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:44.282 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:26:44.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:26:44.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:44.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:44.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:45.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:45.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:45.759 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:26:45.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:26:45.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:45.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:46.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:47.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:47.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:47.347 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:26:47.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:26:47.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:47.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:48.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:49.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:49.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:49.023 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:26:49.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:26:49.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:49.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:49.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:50.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:50.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:50.544 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:26:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:26:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:50.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:51.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:52.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:52.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:52.385 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:26:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:26:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:52.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:53.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:54.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:54.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:54.172 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:26:54.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:26:54.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:54.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:54.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:55.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:55.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:55.645 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:26:55.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:26:55.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:55.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:56.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:57.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:57.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:57.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:26:57.313 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:26:57.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:57.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:57.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:58.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:58.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:58.959 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:26:58.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:26:58.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:58.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:59.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:00.705 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:27:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:27:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:01.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:02.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:02.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:02.439 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:27:02.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:27:02.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:02.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:02.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:03.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:03.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:03.944 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:27:03.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:27:03.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:03.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:04.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:05.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:05.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:05.920 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:27:05.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:27:05.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:05.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:06.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:07.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:07.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:07.669 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:27:07.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:27:07.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:07.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:08.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:09.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:09.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:09.561 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:27:09.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:27:09.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:09.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:10.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:11.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:11.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:11.986 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:27:11.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:27:11.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:11.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:12.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:13.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:13.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:13.921 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:27:13.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:27:13.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:13.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:14.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:15.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:15.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:15.791 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:27:15.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:27:15.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:15.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:27:16.221 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4127136024,v1:127.0.0.1:6811/4127136024] [v2:127.0.0.1:6812/4127136024,v1:127.0.0.1:6813/4127136024] exists,up b19a6f5d-140b-464a-9622-b95f07512e9c 2024-04-25T08:27:16.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:27:16.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:27:16.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:27:16.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:27:16.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:27:16.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:27:16.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:27:16.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:27:16.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:27:16.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:27:16.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:27:16.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:27:16.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:27:16.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1acd870-87c3-4c6d-ac96-45d9f6136107 2024-04-25T08:27:16.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f1acd870-87c3-4c6d-ac96-45d9f6136107' 2024-04-25T08:27:16.229 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 f1acd870-87c3-4c6d-ac96-45d9f6136107 2024-04-25T08:27:16.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:27:16.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkEypmA1aoDhAAA1zNlgOO508B2XGQnT3l4Q== 2024-04-25T08:27:16.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkEypmA1aoDhAAA1zNlgOO508B2XGQnT3l4Q=="}' 2024-04-25T08:27:16.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1acd870-87c3-4c6d-ac96-45d9f6136107 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:27:16.815 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:27:16.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:27:16.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkEypmA1aoDhAAA1zNlgOO508B2XGQnT3l4Q== --osd-uuid f1acd870-87c3-4c6d-ac96-45d9f6136107 2024-04-25T08:27:16.867 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:16.863+0000 7fa918ac17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:16.869 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:16.867+0000 7fa918ac17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:16.871 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:16.867+0000 7fa918ac17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:16.871 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:16.867+0000 7fa918ac17c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:27:16.871 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:16.871+0000 7fa918ac17c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:27:16.871 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:16.871+0000 7fa918ac17c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:27:16.949 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:16.947+0000 7fa918ac17c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:27:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:27:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:27:25.445 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:27:25.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:27:25.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:27:26.033 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:27:26.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:27:26.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:27:26.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:27:26.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:27:26.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:27:26.064 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:26.063+0000 7f550717a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:26.066 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:26.063+0000 7f550717a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:26.068 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:26.067+0000 7f550717a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:26.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:27:26.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:27:26.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:27:26.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:27:26.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:27:26.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:26.451 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:27:26.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:27:26.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:26.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:26.616 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:26.615+0000 7f550717a7c0 -1 Falling back to public interface 2024-04-25T08:27:26.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:27.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:27.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:27.954 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:27:27.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:27:27.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:27.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:28.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:29.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:29.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:29.357 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:27:29.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:27:29.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:29.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:29.623 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:27:29.623+0000 7f550717a7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:27:29.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:30.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:30.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:30.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:27:30.942 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:27:30.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:30.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:31.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:32.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:32.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:32.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:27:32.529 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:27:32.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:32.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:33.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:34.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:34.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:34.178 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:27:34.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:27:34.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:34.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:34.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:35.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:35.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:35.641 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:27:35.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:27:35.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:35.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:36.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:37.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:37.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:37.285 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:27:37.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:27:37.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:37.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:37.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:38.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:38.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:38.754 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:27:38.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:27:38.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:38.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:39.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:40.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:40.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:40.393 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:27:40.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:27:40.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:40.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:41.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:42.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:27:42.018 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:27:42.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:42.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:42.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:43.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:43.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:43.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:27:43.595 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:27:43.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:43.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:44.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:45.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:45.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:45.250 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:27:45.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:27:45.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:45.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:45.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:46.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:46.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:46.758 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:27:46.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:27:46.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:46.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:47.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:48.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:48.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:48.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:27:48.312 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:27:48.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:48.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:48.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:49.767 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:27:49.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:27:49.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:49.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:50.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:51.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:51.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:51.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:27:51.736 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:27:51.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:51.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:52.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:53.738 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:27:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:27:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:54.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:55.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:55.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:55.767 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:27:55.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:27:55.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:55.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:56.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:57.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:57.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:57.935 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:27:57.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:27:57.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:57.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:27:58.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:59.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:59.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:59.620 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:27:59.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:27:59.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:59.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:28:00.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:01.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:01.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:01.385 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:28:01.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:28:01.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:01.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:28:02.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:03.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:03.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:03.323 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:28:03.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:28:03.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:03.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:28:03.754 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/85000885,v1:127.0.0.1:6819/85000885] [v2:127.0.0.1:6820/85000885,v1:127.0.0.1:6821/85000885] exists,up f1acd870-87c3-4c6d-ac96-45d9f6136107 2024-04-25T08:28:03.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:28:03.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:28:03.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:28:03.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:28:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:28:03.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:28:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:28:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:28:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:28:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:28:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:28:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:28:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:28:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:28:03.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:28:03.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47916d1e-dad1-47c7-865f-2c55ad49929a 2024-04-25T08:28:03.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 47916d1e-dad1-47c7-865f-2c55ad49929a' 2024-04-25T08:28:03.761 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 47916d1e-dad1-47c7-865f-2c55ad49929a 2024-04-25T08:28:03.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:28:03.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATFCpmqWrCLhAAbqgrheO03Qmi5/1sN1bYRA== 2024-04-25T08:28:03.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATFCpmqWrCLhAAbqgrheO03Qmi5/1sN1bYRA=="}' 2024-04-25T08:28:03.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47916d1e-dad1-47c7-865f-2c55ad49929a -i td/osd-recovery-stats/3/new.json 2024-04-25T08:28:04.449 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:28:04.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:28:04.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATFCpmqWrCLhAAbqgrheO03Qmi5/1sN1bYRA== --osd-uuid 47916d1e-dad1-47c7-865f-2c55ad49929a 2024-04-25T08:28:04.500 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:04.498+0000 7f34521c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:04.502 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:04.498+0000 7f34521c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:04.502+0000 7f34521c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:04.502+0000 7f34521c07c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:28:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:04.502+0000 7f34521c07c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:28:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:04.502+0000 7f34521c07c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:28:04.566 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:04.562+0000 7f34521c07c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:28:12.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:28:12.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:28:12.513 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T08:28:12.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:28:12.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:28:13.101 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T08:28:13.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:28:13.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:28:13.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:28:13.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:28:13.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:28:13.129 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:13.130+0000 7fdaaef0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:13.131 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:13.134+0000 7fdaaef0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:13.133 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:13.134+0000 7fdaaef0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:13.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:28:13.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:28:13.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:28:13.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:28:13.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:28:13.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:13.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:28:13.527 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:28:13.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:13.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:13.677 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:13.678+0000 7fdaaef0a7c0 -1 Falling back to public interface 2024-04-25T08:28:13.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:14.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:14.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:14.960 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:28:14.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:28:14.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:14.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:15.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:16.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:16.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:16.370 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:28:16.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:28:16.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:16.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:16.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:16.454+0000 7fdaaef0a7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:28:16.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:17.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:17.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:17.918 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:28:17.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:28:17.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:17.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:18.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:19.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:19.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:19.343 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:28:19.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:28:19.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:19.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:19.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:20.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:20.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:20.820 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:28:20.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:28:20.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:20.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:21.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:22.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:22.436 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:28:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:28:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:22.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:22.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:23.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:23.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:23.942 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:28:23.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:28:23.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:23.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:24.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:25.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:25.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:25.375 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:28:25.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:28:25.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:25.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:25.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:26.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:26.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:26.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:28:26.993 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:28:26.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:26.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:27.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:28.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:28.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:28.609 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:28:28.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:28:28.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:28.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:29.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:30.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:30.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:30.070 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:28:30.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:28:30.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:30.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:30.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:31.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:31.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:31.775 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:28:31.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:28:31.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:31.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:32.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:33.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:28:33.244 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:28:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:33.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:34.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:34.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:34.781 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:28:34.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:28:34.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:34.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:35.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:36.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:36.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:36.308 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:28:36.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:28:36.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:36.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:37.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:38.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:38.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:38.295 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:28:38.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:28:38.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:38.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:39.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:40.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:40.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:40.075 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:28:40.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:28:40.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:40.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:41.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:42.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:42.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:42.269 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:28:42.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:28:42.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:42.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:43.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:44.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:44.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:44.367 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:28:44.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:28:44.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:44.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:45.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:46.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:46.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:46.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:28:46.152 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:28:46.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:46.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:46.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:47.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:47.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:47.802 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:28:47.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:28:47.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:47.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:48.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:49.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:49.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:49.791 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:28:49.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:28:49.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:49.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:28:50.239 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/289241514,v1:127.0.0.1:6827/289241514] [v2:127.0.0.1:6828/289241514,v1:127.0.0.1:6829/289241514] exists,up 47916d1e-dad1-47c7-865f-2c55ad49929a 2024-04-25T08:28:50.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:28:50.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:28:50.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:28:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:28:50.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:28:50.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:28:50.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e782c47f-abd3-43c7-bb48-d32a00db4118 2024-04-25T08:28:50.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 e782c47f-abd3-43c7-bb48-d32a00db4118' 2024-04-25T08:28:50.247 INFO:tasks.workunit.client.0.smithi088.stdout:add osd4 e782c47f-abd3-43c7-bb48-d32a00db4118 2024-04-25T08:28:50.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:28:50.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCFCpmurPJDxAAMIyzlqczQ7Z5EUKyRBpcTg== 2024-04-25T08:28:50.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCFCpmurPJDxAAMIyzlqczQ7Z5EUKyRBpcTg=="}' 2024-04-25T08:28:50.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e782c47f-abd3-43c7-bb48-d32a00db4118 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:28:50.837 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:28:50.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:28:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCFCpmurPJDxAAMIyzlqczQ7Z5EUKyRBpcTg== --osd-uuid e782c47f-abd3-43c7-bb48-d32a00db4118 2024-04-25T08:28:50.891 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:50.893+0000 7f166f2d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:50.893 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:50.893+0000 7f166f2d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:50.894 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:50.897+0000 7f166f2d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:50.895 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:50.897+0000 7f166f2d87c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:28:50.895 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:50.897+0000 7f166f2d87c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:28:50.895 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:50.897+0000 7f166f2d87c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:28:50.983 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:50.985+0000 7f166f2d87c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:28:59.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:28:59.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:28:59.116 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd4 key to auth repository 2024-04-25T08:28:59.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:28:59.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:28:59.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:28:59.682 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.4 2024-04-25T08:28:59.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:28:59.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:28:59.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:28:59.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:28:59.713 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:59.713+0000 7ff3457977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:59.715 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:59.717+0000 7ff3457977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:59.716 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:28:59.717+0000 7ff3457977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:00.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:00.268 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:00.269+0000 7ff3457977c0 -1 Falling back to public interface 2024-04-25T08:29:00.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:01.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:01.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:01.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:29:01.543 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:29:01.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:01.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:01.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:02.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:02.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:02.960 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:29:02.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:29:02.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:02.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:03.197 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:03.197+0000 7ff3457977c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:29:03.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:04.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:04.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:04.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:29:04.516 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:29:04.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:04.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:04.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:05.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:05.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:05.996 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:29:05.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:29:05.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:05.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:06.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:07.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:07.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:07.576 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:29:07.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:29:07.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:07.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:08.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:09.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:09.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:09.176 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:29:09.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:29:09.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:09.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:09.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:10.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:10.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:10.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:29:10.893 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:29:10.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:10.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:11.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:12.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:12.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:12.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:29:12.492 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:29:12.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:12.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:12.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:13.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:13.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:13.952 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:29:13.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:29:13.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:13.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:14.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:15.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:15.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:15.584 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:29:15.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:29:15.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:15.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:16.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:17.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:17.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:17.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:29:17.107 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:29:17.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:17.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:17.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:18.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:18.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:18.601 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:29:18.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:29:18.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:18.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:20.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:20.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:20.190 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:29:20.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:29:20.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:20.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:20.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:21.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:21.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:21.671 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:29:21.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:29:21.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:21.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:22.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:23.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:23.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:23.423 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:29:23.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:29:23.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:23.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:25.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:25.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:25.158 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:29:25.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:29:25.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:25.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:25.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:26.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:26.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:26.683 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:29:26.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:29:26.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:26.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:27.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:28.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:28.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:28.613 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:29:28.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:29:28.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:28.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:29.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:30.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:30.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:30.613 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:29:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:29:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:30.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:31.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:32.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:32.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:32.665 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:29:32.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:29:32.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:32.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:34.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:34.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:34.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:29:34.543 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:29:34.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:34.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:35.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:36.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:36.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:36.371 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:29:36.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:29:36.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:36.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:29:36.801 INFO:tasks.workunit.client.0.smithi088.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1091089658,v1:127.0.0.1:6835/1091089658] [v2:127.0.0.1:6836/1091089658,v1:127.0.0.1:6837/1091089658] exists,up e782c47f-abd3-43c7-bb48-d32a00db4118 2024-04-25T08:29:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:29:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:29:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:29:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-04-25T08:29:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:29:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:29:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:29:36.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:29:36.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:29:36.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:29:36.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:29:36.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:29:36.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:29:36.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1e03c3be-fc7d-44a5-ac31-aa8b537b833a 2024-04-25T08:29:36.807 INFO:tasks.workunit.client.0.smithi088.stdout:add osd5 1e03c3be-fc7d-44a5-ac31-aa8b537b833a 2024-04-25T08:29:36.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 1e03c3be-fc7d-44a5-ac31-aa8b537b833a' 2024-04-25T08:29:36.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:29:36.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwFCpmi7aSMRAAA1s7ur+dnAjcR5pClLnquQ== 2024-04-25T08:29:36.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwFCpmi7aSMRAAA1s7ur+dnAjcR5pClLnquQ=="}' 2024-04-25T08:29:36.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1e03c3be-fc7d-44a5-ac31-aa8b537b833a -i td/osd-recovery-stats/5/new.json 2024-04-25T08:29:37.393 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:29:37.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:29:37.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwFCpmi7aSMRAAA1s7ur+dnAjcR5pClLnquQ== --osd-uuid 1e03c3be-fc7d-44a5-ac31-aa8b537b833a 2024-04-25T08:29:37.437 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:37.436+0000 7fea0209a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:37.439 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:37.440+0000 7fea0209a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:37.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:37.440+0000 7fea0209a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:37.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:37.440+0000 7fea0209a7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:29:37.441 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:37.440+0000 7fea0209a7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:29:37.441 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:37.440+0000 7fea0209a7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:29:37.535 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:37.536+0000 7fea0209a7c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:29:46.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:29:46.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:29:46.335 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd5 key to auth repository 2024-04-25T08:29:46.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:29:46.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:29:46.897 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.5 2024-04-25T08:29:46.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:29:46.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:29:46.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:29:46.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:29:46.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:29:46.928 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:46.928+0000 7fc01cfe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:46.972 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:46.972+0000 7fc01cfe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:46.974 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:46.976+0000 7fc01cfe37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:47.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:29:47.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:29:47.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:29:47.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:29:47.303 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:29:47.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:29:47.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:47.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:29:47.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:47.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:29:47.532 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:47.532+0000 7fc01cfe37c0 -1 Falling back to public interface 2024-04-25T08:29:47.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:48.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:48.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:48.757 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:29:48.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:29:48.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:48.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:29:49.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:50.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:50.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:50.196 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:29:50.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:29:50.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:50.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:29:50.384 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:29:50.384+0000 7fc01cfe37c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:29:50.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:51.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:51.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:51.702 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:29:51.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:29:51.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:51.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:29:52.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:53.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:53.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:53.289 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:29:53.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:29:53.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:53.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:29:54.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:55.019 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:29:55.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:29:55.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:55.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:29:55.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:56.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:56.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:56.537 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:29:56.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:29:56.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:56.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:29:56.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:57.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:57.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:57.977 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:29:57.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:29:57.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:57.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:29:58.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:59.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:59.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:59.583 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:29:59.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:29:59.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:59.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:00.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:01.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:01.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:01.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:30:01.102 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:30:01.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:01.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:01.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:02.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:02.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:02.695 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:30:02.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:30:02.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:02.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:03.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:04.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:04.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:04.124 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:30:04.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:30:04.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:04.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:04.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:05.701 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:30:05.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:05.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:05.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:30:05.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:05.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:06.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:07.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:07.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:07.137 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:30:07.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:30:07.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:07.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:07.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:08.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:08.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:08.739 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:30:08.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:30:08.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:08.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:09.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:10.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:10.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:10.312 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:30:10.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:30:10.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:10.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:11.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:12.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:12.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:12.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:30:12.161 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:30:12.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:12.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:12.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:13.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:13.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:13.955 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:30:13.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:30:13.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:13.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:14.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:15.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:15.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:15.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:30:15.920 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:30:15.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:15.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:16.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:17.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:17.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:17.796 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:30:17.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:30:17.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:17.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:18.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:19.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:19.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:19.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:30:19.570 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:30:19.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:19.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:20.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:21.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:21.950 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:30:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:30:21.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:21.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:22.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:23.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:23.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:23.827 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:30:23.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:30:23.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:23.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:30:24.252 INFO:tasks.workunit.client.0.smithi088.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1516653364,v1:127.0.0.1:6843/1516653364] [v2:127.0.0.1:6844/1516653364,v1:127.0.0.1:6845/1516653364] exists,up 1e03c3be-fc7d-44a5-ac31-aa8b537b833a 2024-04-25T08:30:24.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:30:24.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:30:24.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:30:24.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2024-04-25T08:30:24.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2024-04-25T08:30:24.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 replicated 2024-04-25T08:30:25.142 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T08:30:25.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:30:26.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-04-25T08:30:26.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:30:26.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:30:26.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:30:26.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:30:26.163 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:30:26.163 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:30:26.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:30:26.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:30:26.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:30:26.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:30:26.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:30:26.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:30:26.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:30:26.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:30:26.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:30:26.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:30:26.726 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:30:26.726 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:30:26.726 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:30:26.726 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:30:26.726 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:30:26.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:30:26.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:30:26.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:30:26.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836538 2024-04-25T08:30:26.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836538 2024-04-25T08:30:26.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538' 2024-04-25T08:30:26.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:30:26.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:30:27.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673009 2024-04-25T08:30:27.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673009 2024-04-25T08:30:27.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673009' 2024-04-25T08:30:27.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:30:27.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:30:27.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509482 2024-04-25T08:30:27.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509482 2024-04-25T08:30:27.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673009 2-64424509482' 2024-04-25T08:30:27.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:30:27.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:30:27.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345948 2024-04-25T08:30:27.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345948 2024-04-25T08:30:27.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673009 2-64424509482 3-85899345948' 2024-04-25T08:30:27.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:30:27.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:30:27.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182417 2024-04-25T08:30:27.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182417 2024-04-25T08:30:27.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673009 2-64424509482 3-85899345948 4-107374182417' 2024-04-25T08:30:27.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:30:27.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:30:28.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018894 2024-04-25T08:30:28.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018894 2024-04-25T08:30:28.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673009 2-64424509482 3-85899345948 4-107374182417 5-128849018894' 2024-04-25T08:30:28.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:30:28.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836538 2024-04-25T08:30:28.098 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:30:28.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:30:28.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836538 2024-04-25T08:30:28.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:30:28.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836538 2024-04-25T08:30:28.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-04-25T08:30:28.101 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836538 2024-04-25T08:30:28.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:30:28.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836536 -lt 21474836538 2024-04-25T08:30:28.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:30:29.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:30:29.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:30:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836538 -lt 21474836538 2024-04-25T08:30:29.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:30:29.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673009 2024-04-25T08:30:29.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:30:29.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:30:29.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673009 2024-04-25T08:30:29.902 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:30:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673009 2024-04-25T08:30:29.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2024-04-25T08:30:29.904 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673009 2024-04-25T08:30:29.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:30:30.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673009 -lt 42949673009 2024-04-25T08:30:30.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:30:30.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509482 2024-04-25T08:30:30.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:30:30.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:30:30.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509482 2024-04-25T08:30:30.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:30:30.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509482 2024-04-25T08:30:30.413 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509482 2024-04-25T08:30:30.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-04-25T08:30:30.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:30:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509483 -lt 64424509482 2024-04-25T08:30:30.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:30:30.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345948 2024-04-25T08:30:30.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:30:30.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:30:30.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345948 2024-04-25T08:30:30.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:30:30.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345948 2024-04-25T08:30:30.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345948' 2024-04-25T08:30:30.889 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345948 2024-04-25T08:30:30.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:30:31.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345949 -lt 85899345948 2024-04-25T08:30:31.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:30:31.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182417 2024-04-25T08:30:31.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:30:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:30:31.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182417 2024-04-25T08:30:31.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:30:31.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182417 2024-04-25T08:30:31.290 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182417 2024-04-25T08:30:31.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2024-04-25T08:30:31.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:30:31.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182417 -lt 107374182417 2024-04-25T08:30:31.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:30:31.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:30:31.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018894 2024-04-25T08:30:31.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:30:31.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018894 2024-04-25T08:30:31.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:30:31.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018894 2024-04-25T08:30:31.747 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 128849018894 2024-04-25T08:30:31.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2024-04-25T08:30:31.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:30:32.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018895 -lt 128849018894 2024-04-25T08:30:32.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:30:32.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:32.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:32.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:30:32.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:32.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:32.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:32.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:32.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:32.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:32.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:33.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:33.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:33.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:33.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:33.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:30:33.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:30:33.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:30:33.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-04-25T08:30:33.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:33.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-04-25T08:30:33.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:33.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-04-25T08:30:34.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:34.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-04-25T08:30:34.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:34.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-04-25T08:30:34.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:34.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-04-25T08:30:34.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:34.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-04-25T08:30:34.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:34.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-04-25T08:30:35.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:35.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-04-25T08:30:35.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:35.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-04-25T08:30:35.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:35.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-04-25T08:30:35.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:35.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-04-25T08:30:35.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:35.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-04-25T08:30:35.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:35.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-04-25T08:30:36.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:36.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-04-25T08:30:36.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:36.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-04-25T08:30:36.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:36.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-04-25T08:30:36.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:36.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-04-25T08:30:36.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:36.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-04-25T08:30:36.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:36.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-04-25T08:30:36.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:36.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-04-25T08:30:37.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:37.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-04-25T08:30:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-04-25T08:30:37.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:37.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-04-25T08:30:37.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:37.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-04-25T08:30:37.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-04-25T08:30:38.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:38.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-04-25T08:30:38.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:38.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-04-25T08:30:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:38.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-04-25T08:30:38.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:38.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-04-25T08:30:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:38.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-04-25T08:30:38.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:38.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-04-25T08:30:38.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:38.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-04-25T08:30:39.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:39.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-04-25T08:30:39.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:39.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-04-25T08:30:39.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:39.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-04-25T08:30:39.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:39.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-04-25T08:30:39.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:39.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-04-25T08:30:39.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:39.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-04-25T08:30:40.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:40.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-04-25T08:30:40.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:40.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-04-25T08:30:40.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:40.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-04-25T08:30:40.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:40.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-04-25T08:30:40.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:40.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-04-25T08:30:40.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:40.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-04-25T08:30:41.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:41.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-04-25T08:30:41.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:41.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-04-25T08:30:41.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:41.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-04-25T08:30:41.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:41.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-04-25T08:30:41.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:41.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-04-25T08:30:41.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:41.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-04-25T08:30:42.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:42.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-04-25T08:30:42.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:42.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-04-25T08:30:42.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:42.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-04-25T08:30:42.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:42.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-04-25T08:30:42.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:42.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-04-25T08:30:42.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:42.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-04-25T08:30:43.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:43.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-04-25T08:30:43.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:43.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-04-25T08:30:43.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:43.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-04-25T08:30:43.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:43.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-04-25T08:30:43.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:43.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-04-25T08:30:43.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:43.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-04-25T08:30:44.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:44.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-04-25T08:30:44.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:44.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-04-25T08:30:44.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:44.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-04-25T08:30:44.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:44.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-04-25T08:30:44.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:44.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-04-25T08:30:44.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:44.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-04-25T08:30:44.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:44.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-04-25T08:30:45.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:45.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-04-25T08:30:45.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:45.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-04-25T08:30:45.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:45.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-04-25T08:30:45.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:45.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-04-25T08:30:45.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:45.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-04-25T08:30:45.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:45.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-04-25T08:30:46.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:46.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-04-25T08:30:46.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:46.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-04-25T08:30:46.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:46.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-04-25T08:30:46.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:46.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-04-25T08:30:46.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:46.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-04-25T08:30:46.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:46.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-04-25T08:30:47.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:47.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-04-25T08:30:47.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:47.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-04-25T08:30:47.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:47.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-04-25T08:30:47.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:47.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-04-25T08:30:47.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:47.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-04-25T08:30:47.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:47.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-04-25T08:30:48.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:48.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-04-25T08:30:48.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:48.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-04-25T08:30:48.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:48.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-04-25T08:30:48.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:48.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-04-25T08:30:48.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:48.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-04-25T08:30:48.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:48.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-04-25T08:30:49.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:49.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-04-25T08:30:49.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:49.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-04-25T08:30:49.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:49.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-04-25T08:30:49.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:49.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-04-25T08:30:49.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:49.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-04-25T08:30:49.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:49.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-04-25T08:30:50.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:50.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-04-25T08:30:50.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:50.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-04-25T08:30:50.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:50.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-04-25T08:30:50.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:50.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-04-25T08:30:50.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:50.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-04-25T08:30:50.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:50.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-04-25T08:30:51.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:51.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-04-25T08:30:51.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:51.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-04-25T08:30:51.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:51.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-04-25T08:30:51.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:51.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-04-25T08:30:51.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:51.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-04-25T08:30:51.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:51.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-04-25T08:30:52.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:52.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-04-25T08:30:52.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:52.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-04-25T08:30:52.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:52.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-04-25T08:30:52.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:52.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-04-25T08:30:52.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:52.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-04-25T08:30:52.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:52.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-04-25T08:30:53.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:53.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-04-25T08:30:53.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:53.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-04-25T08:30:53.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:53.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-04-25T08:30:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-04-25T08:30:53.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:53.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-04-25T08:30:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-04-25T08:30:54.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:54.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-04-25T08:30:54.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:54.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-04-25T08:30:54.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:54.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-04-25T08:30:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:54.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-04-25T08:30:54.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:54.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-04-25T08:30:54.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:54.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-04-25T08:30:54.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:54.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-04-25T08:30:55.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:55.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-04-25T08:30:55.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:55.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-04-25T08:30:55.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:55.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-04-25T08:30:55.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:55.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-04-25T08:30:55.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:55.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-04-25T08:30:56.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:56.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-04-25T08:30:56.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:56.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-04-25T08:30:56.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:56.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-04-25T08:30:56.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:56.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-04-25T08:30:56.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:56.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-04-25T08:30:56.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:56.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-04-25T08:30:56.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:56.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-04-25T08:30:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:57.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-04-25T08:30:57.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:57.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-04-25T08:30:57.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:57.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-04-25T08:30:57.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:57.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-04-25T08:30:57.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:57.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-04-25T08:30:57.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:57.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-04-25T08:30:58.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:58.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-04-25T08:30:58.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:58.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-04-25T08:30:58.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:58.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-04-25T08:30:58.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:58.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-04-25T08:30:58.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:58.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-04-25T08:30:58.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:58.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-04-25T08:30:59.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:59.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-04-25T08:30:59.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:59.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-04-25T08:30:59.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:59.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-04-25T08:30:59.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:59.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-04-25T08:30:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:59.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-04-25T08:30:59.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:59.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-04-25T08:30:59.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:30:59.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-04-25T08:31:00.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:00.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-04-25T08:31:00.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:00.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-04-25T08:31:00.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:00.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-04-25T08:31:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:00.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-04-25T08:31:00.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:00.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-04-25T08:31:00.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:00.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-04-25T08:31:01.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:01.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-04-25T08:31:01.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:01.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-04-25T08:31:01.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:01.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-04-25T08:31:01.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:01.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-04-25T08:31:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:01.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-04-25T08:31:01.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:01.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-04-25T08:31:02.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:02.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-04-25T08:31:02.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:02.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-04-25T08:31:02.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:02.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-04-25T08:31:02.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:02.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-04-25T08:31:02.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:02.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-04-25T08:31:02.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:02.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-04-25T08:31:03.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:03.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-04-25T08:31:03.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:03.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-04-25T08:31:03.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:03.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-04-25T08:31:03.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:03.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-04-25T08:31:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:03.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-04-25T08:31:03.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:03.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-04-25T08:31:04.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:04.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-04-25T08:31:04.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:04.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-04-25T08:31:04.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:04.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-04-25T08:31:04.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:04.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-04-25T08:31:04.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:04.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-04-25T08:31:04.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:04.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-04-25T08:31:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-04-25T08:31:05.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:05.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-04-25T08:31:05.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:05.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-04-25T08:31:05.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:05.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-04-25T08:31:06.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:06.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-04-25T08:31:06.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:06.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-04-25T08:31:06.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:06.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-04-25T08:31:06.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:06.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-04-25T08:31:06.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:06.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-04-25T08:31:06.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-04-25T08:31:06.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:31:06.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:31:06.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:31:06.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:31:07.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-04-25T08:31:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-04-25T08:31:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:31:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:31:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:31:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:31:07.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-04-25T08:31:07.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-04-25T08:31:07.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:31:07.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:31:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:31:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:31:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:31:07.755 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:31:07.755 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:31:08.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:31:08.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:31:08.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:31:08.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-04-25T08:31:08.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-04-25T08:31:09.522 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is set 2024-04-25T08:31:09.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:31:09.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 326918 2024-04-25T08:31:09.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-04-25T08:31:10.047 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T08:31:10.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-04-25T08:31:11.111 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T08:31:11.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-04-25T08:31:12.266 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is unset 2024-04-25T08:31:12.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-04-25T08:31:12.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:31:12.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:31:12.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:31:12.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:31:12.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:31:13.216 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:31:13.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-04-25T08:31:15.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-04-25T08:31:15.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:31:15.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:31:15.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:31:15.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:31:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:31:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:31:15.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:31:15.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:31:15.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:31:15.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:31:15.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:31:15.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:31:15.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:31:15.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:31:15.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:31:15.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:31:15.897 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:31:15.897 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:31:15.897 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:31:15.897 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:31:15.898 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:31:15.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:31:15.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:15.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:31:16.120 INFO:tasks.workunit.client.0.smithi088.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:31:16.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:31:16.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:31:16.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:31:16.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:16.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:31:16.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673021 2024-04-25T08:31:16.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673021 2024-04-25T08:31:16.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673021' 2024-04-25T08:31:16.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:16.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:31:16.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509494 2024-04-25T08:31:16.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509494 2024-04-25T08:31:16.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673021 2-64424509494' 2024-04-25T08:31:16.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:16.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:31:16.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345960 2024-04-25T08:31:16.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345960 2024-04-25T08:31:16.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673021 2-64424509494 3-85899345960' 2024-04-25T08:31:16.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:16.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:31:16.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182428 2024-04-25T08:31:16.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182428 2024-04-25T08:31:16.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673021 2-64424509494 3-85899345960 4-107374182428' 2024-04-25T08:31:16.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:16.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:31:17.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018906 2024-04-25T08:31:17.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018906 2024-04-25T08:31:17.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673021 2-64424509494 3-85899345960 4-107374182428 5-128849018906' 2024-04-25T08:31:17.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:17.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673021 2024-04-25T08:31:17.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:17.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:31:17.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673021 2024-04-25T08:31:17.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:17.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673021 2024-04-25T08:31:17.033 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673021 2024-04-25T08:31:17.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673021' 2024-04-25T08:31:17.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:31:17.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673021 -lt 42949673021 2024-04-25T08:31:17.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:17.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509494 2024-04-25T08:31:17.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:17.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:31:17.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509494 2024-04-25T08:31:17.609 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:17.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509494 2024-04-25T08:31:17.610 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509494 2024-04-25T08:31:17.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-04-25T08:31:17.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:31:18.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509494 -lt 64424509494 2024-04-25T08:31:18.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:18.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:18.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345960 2024-04-25T08:31:18.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:31:18.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345960 2024-04-25T08:31:18.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:18.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345960 2024-04-25T08:31:18.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345960' 2024-04-25T08:31:18.058 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345960 2024-04-25T08:31:18.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:31:18.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345960 -lt 85899345960 2024-04-25T08:31:18.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:18.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182428 2024-04-25T08:31:18.550 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:18.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:31:18.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182428 2024-04-25T08:31:18.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:18.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182428 2024-04-25T08:31:18.553 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182428 2024-04-25T08:31:18.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182428' 2024-04-25T08:31:18.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:31:18.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182428 -lt 107374182428 2024-04-25T08:31:18.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:18.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018906 2024-04-25T08:31:18.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:18.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:31:18.974 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:18.974 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018906 2024-04-25T08:31:18.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018906 2024-04-25T08:31:18.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018906' 2024-04-25T08:31:18.975 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 128849018906 2024-04-25T08:31:18.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:31:19.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018906 -lt 128849018906 2024-04-25T08:31:19.621 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:31:19.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:19.622 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:20.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:31:20.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:20.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:20.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:20.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:20.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:20.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:20.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:20.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:20.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:20.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:20.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:21.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:21.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:31:21.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:31:21.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:31:21.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:21.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:21.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:21.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:21.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:21.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:21.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:21.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:21.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:21.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:22.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:22.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:22.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:22.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:22.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:22.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:22.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:22.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:22.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:22.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:22.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:22.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:22.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:23.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:31:23.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:31:23.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:23.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:23.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:23.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:23.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:23.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:23.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:23.447 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:23.447 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:23.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:23.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:23.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:23.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:24.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:24.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:24.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:24.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:24.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:24.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:24.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:24.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:24.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:25.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:31:25.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:31:25.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:25.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:25.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:25.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:25.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:25.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:25.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:25.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:25.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:25.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:25.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:25.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:25.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:25.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:25.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:26.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:26.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:26.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:26.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:31:26.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:31:26.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:26.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:26.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:26.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:26.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:26.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:26.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:26.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:26.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:26.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:26.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:27.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:27.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:27.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:27.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:27.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:27.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:27.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:27.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:27.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:27.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:27.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:27.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:27.989 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:28.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:31:28.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:31:28.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:28.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:28.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:28.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:28.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:28.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:28.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:28.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:28.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:28.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:28.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:29.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:29.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:29.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:29.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:29.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:29.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:29.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:29.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:29.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:29.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:29.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:29.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:29.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:30.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:31:30.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:31:30.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:30.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:30.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:30.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:30.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:30.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:30.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:30.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:30.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:30.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:30.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:30.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:30.959 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:30.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:30.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:31.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:31.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:31.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:31.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:31.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:31.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:31.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:31.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:31.567 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:32.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-04-25T08:31:32.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-04-25T08:31:32.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:32.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:32.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:32.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:32.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:32.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:32.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:32.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:32.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:32.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:32.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:32.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:32.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:32.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:32.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:33.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:33.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:33.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:33.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:33.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:33.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:33.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:33.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:33.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:33.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:31:33.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:31:33.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:33.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:33.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:33.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:33.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:33.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:33.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:33.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:33.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:34.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:34.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:34.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:34.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:35.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:35.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:35.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:35.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:35.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:31:35.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:31:35.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:35.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:35.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:35.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:35.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:35.942 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:35.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:35.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:35.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:35.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:35.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:36.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:31:36.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:36.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:36.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:31:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:31:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:36.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:36.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:36.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:31:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:31:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:31:37.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:37.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:37.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:37.642 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:37.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:37.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:37.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:37.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:37.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:38.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:38.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:38.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:38.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:31:38.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:31:38.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:31:38.649 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:31:38.649 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:31:38.649 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:31:38.649 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:31:38.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2024-04-25T08:31:38.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:31:38.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:31:38.661 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:31:38.661 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2024-04-25T08:31:38.661 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:31:38.661 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:31:38.661 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:31:38.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:31:38.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:31:38.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:31:38.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:31:38.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:31:38.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:31:38.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:31:38.669 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:31:38.669 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:31:38.669 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:31:38.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:31:38.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:31:38.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:31:38.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:31:38.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:31:38.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:31:38.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:31:38.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:31:38.678 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:31:38.678 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2024-04-25T08:31:38.678 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:31:38.678 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:31:38.678 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:31:38.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:31:38.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:31:38.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:31:38.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:31:38.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:31:38.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:31:38.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:31:38.685 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:31:38.685 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:31:38.685 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:31:38.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:31:38.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:31:39.436 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T08:31:39.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-04-25T08:31:39.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:31:39.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:31:39.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:31:39.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:31:39.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:31:44.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:31:44.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:31:44.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:31:44.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:31:44.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:31:44.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:31:44.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:31:44.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:31:44.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:31:44.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:31:44.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:31:44.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:31:44.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:31:44.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:31:44.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:31:44.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:31:44.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:31:44.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:31:44.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:31:44.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:31:44.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:31:44.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:31:44.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:31:44.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:31:44.837 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:31:44.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:31:44.838 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:31:44.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:31:44.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:31:44.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:31:44.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:31:44.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:31:44.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:31:44.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:31:44.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:31:44.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:31:44.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:31:44.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:31:44.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:31:44.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:31:44.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:31:44.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:31:44.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:31:44.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:31:44.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:31:44.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:31:44.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:31:44.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274961 2024-04-25T08:31:44.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:31:44.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:31:44.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:31:45.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:31:45.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:31:45.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:31:45.370 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:31:45.371 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:31:45.371 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:31:45.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:31:45.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:31:45.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:31:45.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:31:45.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:31:45.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:31:45.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:31:45.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:31:45.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:31:45.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:31:45.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:31:45.626 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:31:45.626 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:31:45.626 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:31:45.626 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:31:45.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:31:45.626 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get fsid 2024-04-25T08:31:45.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:31:45.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:31:45.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:31:45.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:31:45.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:31:45.742 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:31:45.742 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:31:45.742 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:31:45.743 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:31:45.743 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:31:45.743 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:31:45.743 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:31:45.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:31:45.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get mon_host 2024-04-25T08:31:45.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2024-04-25T08:31:45.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:31:45.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:31:45.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:31:45.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:31:45.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:31:45.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:31:46.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:31:46.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:31:46.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:31:46.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:31:46.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:31:46.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:31:46.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:31:46.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:31:46.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:31:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2024-04-25T08:31:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:31:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:31:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:31:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:31:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:31:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:31:46.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:31:46.842 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:31:46.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:31:46.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:31:46.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:31:46.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a61813c3-8e84-44d1-af2c-95a167025638 2024-04-25T08:31:46.844 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 a61813c3-8e84-44d1-af2c-95a167025638 2024-04-25T08:31:46.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a61813c3-8e84-44d1-af2c-95a167025638' 2024-04-25T08:31:46.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:31:46.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyFCpm1MWWMxAAozzhVtGn7RuzbmTTe9ZaDQ== 2024-04-25T08:31:46.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyFCpm1MWWMxAAozzhVtGn7RuzbmTTe9ZaDQ=="}' 2024-04-25T08:31:46.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a61813c3-8e84-44d1-af2c-95a167025638 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:31:47.209 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:31:47.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:31:47.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyFCpm1MWWMxAAozzhVtGn7RuzbmTTe9ZaDQ== --osd-uuid a61813c3-8e84-44d1-af2c-95a167025638 2024-04-25T08:31:47.258 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:47.261+0000 7fdf20c577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:47.310 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:47.313+0000 7fdf20c577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:47.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:47.313+0000 7fdf20c577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:47.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:47.317+0000 7fdf20c577c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:31:47.313 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:47.317+0000 7fdf20c577c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:31:47.313 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:47.317+0000 7fdf20c577c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:31:47.351 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:47.353+0000 7fdf20c577c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:31:55.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:31:55.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:31:55.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:31:55.140 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:31:55.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:31:55.709 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:31:55.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:31:55.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:31:55.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:31:55.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:31:55.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:31:55.744 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:55.745+0000 7f5e2e8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:55.746 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:55.749+0000 7f5e2e8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:55.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:55.749+0000 7f5e2e8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:56.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:31:56.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:31:56.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:31:56.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:31:56.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:31:56.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:56.106 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:31:56.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:31:56.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:56.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:31:56.326 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:56.329+0000 7f5e2e8087c0 -1 Falling back to public interface 2024-04-25T08:31:56.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:57.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:57.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:57.505 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:31:57.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:31:57.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:57.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:31:57.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:58.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:58.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:58.918 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:31:58.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:31:58.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:58.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:31:59.160 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:31:59.161+0000 7f5e2e8087c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:31:59.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:00.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:00.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:00.483 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:32:00.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:32:00.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:00.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:00.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:01.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:01.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:01.934 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:32:01.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:32:01.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:01.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:02.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:03.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:03.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:03.721 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:32:03.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:32:03.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:03.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:04.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:05.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:05.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:05.387 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:32:05.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:32:05.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:05.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:06.841 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:32:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:32:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:07.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:08.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:08.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:08.476 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:32:08.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:32:08.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:08.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:09.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:10.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:10.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:10.004 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:32:10.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:32:10.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:10.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:10.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:11.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:11.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:11.418 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:32:11.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:32:11.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:11.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:12.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:13.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:13.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:13.228 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:32:13.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:32:13.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:13.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:13.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:14.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:14.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:14.794 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:32:14.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:32:14.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:14.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:15.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:16.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:16.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:16.313 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:32:16.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:32:16.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:16.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:16.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:17.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:17.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:17.818 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:32:17.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:32:17.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:17.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:18.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:19.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:19.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:19.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:32:19.394 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:32:19.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:19.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:19.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:20.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:20.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:20.900 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:32:20.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:32:20.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:20.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:21.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:22.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:22.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:22.682 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:32:22.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:32:22.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:22.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:23.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:24.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:24.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:24.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:32:24.511 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:32:24.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:24.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:25.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:26.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:26.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:26.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:32:26.412 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:32:26.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:26.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:27.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:28.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:28.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:28.335 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:32:28.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:32:28.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:28.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:29.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:30.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:30.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:30.284 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:32:30.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:32:30.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:30.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:31.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:32.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:32.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:32.008 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:32:32.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:32:32.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:32.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:32.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:33.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:33.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:33.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-04-25T08:32:33.419 INFO:tasks.workunit.client.0.smithi088.stdout:23 2024-04-25T08:32:33.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:33.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:32:33.880 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3564036611,v1:127.0.0.1:6803/3564036611] [v2:127.0.0.1:6804/3564036611,v1:127.0.0.1:6805/3564036611] exists,up a61813c3-8e84-44d1-af2c-95a167025638 2024-04-25T08:32:33.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:32:33.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:32:33.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:32:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:32:33.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:32:33.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:32:33.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:32:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:32:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:32:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:32:33.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:32:33.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:32:33.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:32:33.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc8362a8-b823-4651-a1d8-b5ab50b54a10 2024-04-25T08:32:33.888 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 fc8362a8-b823-4651-a1d8-b5ab50b54a10 2024-04-25T08:32:33.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fc8362a8-b823-4651-a1d8-b5ab50b54a10' 2024-04-25T08:32:33.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:32:33.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhFSpmaDRoNhAAz3fnXOuqQJuP7nMgcbk+qg== 2024-04-25T08:32:33.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhFSpmaDRoNhAAz3fnXOuqQJuP7nMgcbk+qg=="}' 2024-04-25T08:32:33.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc8362a8-b823-4651-a1d8-b5ab50b54a10 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:32:34.468 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:32:34.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:32:34.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhFSpmaDRoNhAAz3fnXOuqQJuP7nMgcbk+qg== --osd-uuid fc8362a8-b823-4651-a1d8-b5ab50b54a10 2024-04-25T08:32:34.525 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:34.528+0000 7fc61d13d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:34.571 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:34.572+0000 7fc61d13d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:34.576+0000 7fc61d13d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:34.576+0000 7fc61d13d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:32:34.573 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:34.576+0000 7fc61d13d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:32:34.574 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:34.576+0000 7fc61d13d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:32:34.610 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:34.612+0000 7fc61d13d7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:32:42.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:32:42.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:32:42.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:32:42.603 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:32:42.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:32:43.166 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:32:43.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:32:43.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:32:43.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:32:43.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:32:43.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:32:43.198 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:43.200+0000 7f25ff6b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:43.200 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:43.204+0000 7f25ff6b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:43.201 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:43.204+0000 7f25ff6b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:32:43.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:32:43.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:32:43.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:32:43.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:32:43.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:32:43.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:43.553 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:32:43.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:32:43.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:43.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:43.750 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:43.752+0000 7f25ff6b87c0 -1 Falling back to public interface 2024-04-25T08:32:44.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:45.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:45.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:45.061 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:32:45.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:32:45.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:45.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:45.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:46.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:46.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:46.477 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:32:46.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:32:46.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:46.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:46.751 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:32:46.752+0000 7f25ff6b87c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:32:46.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:47.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:47.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:47.901 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:32:47.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:32:47.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:47.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:48.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:49.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:49.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:49.596 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:32:49.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:32:49.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:49.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:50.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:51.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:51.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:51.174 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:32:51.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:32:51.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:51.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:51.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:52.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:52.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:52.787 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:32:52.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:32:52.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:52.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:53.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:54.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:54.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:54.390 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:32:54.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:32:54.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:54.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:55.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:56.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:56.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:56.064 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:32:56.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:32:56.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:56.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:56.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:57.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:57.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:57.571 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:32:57.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:32:57.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:57.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:58.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:32:59.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:32:59.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:32:59.259 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:32:59.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:32:59.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:32:59.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:32:59.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:00.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:00.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:00.787 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:33:00.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:33:00.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:00.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:01.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:02.267 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:33:02.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:33:02.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:02.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:02.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:03.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:03.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:03.994 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:33:03.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:33:03.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:03.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:04.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:05.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:05.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:05.680 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:33:05.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:33:05.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:05.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:06.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:07.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:07.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:07.181 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:33:07.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:33:07.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:07.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:08.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:09.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:09.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:09.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:33:09.073 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:33:09.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:09.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:09.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:10.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:10.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:10.850 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:33:10.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:33:10.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:10.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:11.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:12.775 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:33:12.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:33:12.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:12.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:13.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:14.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:14.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:14.517 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:33:14.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:33:14.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:14.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:15.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:16.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:16.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:16.462 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:33:16.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:33:16.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:16.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:17.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:18.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:18.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:18.424 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:33:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:33:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:18.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:19.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:20.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:20.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:20.575 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:33:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:33:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:21.006 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2293365849,v1:127.0.0.1:6811/2293365849] [v2:127.0.0.1:6812/2293365849,v1:127.0.0.1:6813/2293365849] exists,up fc8362a8-b823-4651-a1d8-b5ab50b54a10 2024-04-25T08:33:21.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:33:21.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:33:21.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:33:21.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:33:21.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:33:21.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:33:21.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2d04d93-c5b7-42ee-bf1d-4486a6c50e21 2024-04-25T08:33:21.013 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 b2d04d93-c5b7-42ee-bf1d-4486a6c50e21 2024-04-25T08:33:21.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b2d04d93-c5b7-42ee-bf1d-4486a6c50e21' 2024-04-25T08:33:21.013 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:33:21.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRFSpm3pMFAhAAUVXyNfMcgSuLDbGbtYikNw== 2024-04-25T08:33:21.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRFSpm3pMFAhAAUVXyNfMcgSuLDbGbtYikNw=="}' 2024-04-25T08:33:21.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2d04d93-c5b7-42ee-bf1d-4486a6c50e21 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:33:21.561 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:33:21.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:33:21.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRFSpm3pMFAhAAUVXyNfMcgSuLDbGbtYikNw== --osd-uuid b2d04d93-c5b7-42ee-bf1d-4486a6c50e21 2024-04-25T08:33:21.614 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:21.615+0000 7f679b9a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:21.616 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:21.619+0000 7f679b9a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:21.618 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:21.619+0000 7f679b9a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:21.619 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:21.619+0000 7f679b9a27c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:33:21.619 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:21.619+0000 7f679b9a27c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:33:21.619 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:21.619+0000 7f679b9a27c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:33:21.717 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:21.719+0000 7f679b9a27c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:33:30.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:33:30.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:33:30.396 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:33:30.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:33:30.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:33:30.962 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:33:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:33:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:33:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:33:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:33:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:33:30.989 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:30.991+0000 7f4b14a397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:30.991 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:30.991+0000 7f4b14a397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:30.992 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:30.995+0000 7f4b14a397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:31.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:33:31.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:33:31.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:33:31.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:33:31.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:33:31.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:31.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:33:31.361 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:33:31.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:31.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:31.541 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:31.543+0000 7f4b14a397c0 -1 Falling back to public interface 2024-04-25T08:33:31.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:32.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:32.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:32.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:33:32.773 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:33:32.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:32.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:33.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:34.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:34.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:34.203 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:33:34.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:33:34.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:34.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:34.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:34.696 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:33:34.699+0000 7f4b14a397c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:33:35.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:35.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:35.618 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:33:35.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:33:35.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:35.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:36.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:37.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:37.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:37.174 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:33:37.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:33:37.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:37.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:37.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:38.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:38.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:38.668 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:33:38.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:33:38.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:38.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:39.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:40.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:40.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:40.295 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:33:40.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:33:40.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:40.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:40.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:41.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:41.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:41.905 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:33:41.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:33:41.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:41.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:42.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:43.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:43.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:43.455 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:33:43.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:33:43.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:43.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:44.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:45.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:45.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:45.127 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:33:45.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:33:45.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:45.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:45.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:46.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:46.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:46.594 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:33:46.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:33:46.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:46.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:47.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:48.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:48.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:48.197 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:33:48.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:33:48.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:48.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:48.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:49.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:49.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:49.922 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:33:49.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:33:49.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:49.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:50.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:51.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:51.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:51.551 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:33:51.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:33:51.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:51.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:51.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:52.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:52.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:52.975 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:33:52.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:33:52.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:52.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:54.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:54.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:54.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:33:54.440 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:33:54.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:54.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:56.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:56.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:56.204 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:33:56.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:33:56.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:56.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:57.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:58.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:58.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:58.544 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:33:58.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:33:58.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:58.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:33:59.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:00.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:00.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:00.565 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:34:00.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:34:00.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:00.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:01.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:02.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:02.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:02.538 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:34:02.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:34:02.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:02.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:03.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:04.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:04.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:04.744 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:34:04.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:34:04.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:04.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:05.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:06.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:06.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:06.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:34:06.635 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:34:06.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:06.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:07.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:08.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:08.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:08.444 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:34:08.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:34:08.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:08.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:08.879 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1011774440,v1:127.0.0.1:6819/1011774440] [v2:127.0.0.1:6820/1011774440,v1:127.0.0.1:6821/1011774440] exists,up b2d04d93-c5b7-42ee-bf1d-4486a6c50e21 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:34:08.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:34:08.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:34:08.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:34:08.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:34:08.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:34:08.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ff353be-724c-4f0e-a6f1-0dd810c0a08b 2024-04-25T08:34:08.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9ff353be-724c-4f0e-a6f1-0dd810c0a08b' 2024-04-25T08:34:08.885 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 9ff353be-724c-4f0e-a6f1-0dd810c0a08b 2024-04-25T08:34:08.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:34:08.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAFSpmouhGNhAAXLCbaVb84EJrgG5AG3HlyA== 2024-04-25T08:34:08.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAFSpmouhGNhAAXLCbaVb84EJrgG5AG3HlyA=="}' 2024-04-25T08:34:08.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ff353be-724c-4f0e-a6f1-0dd810c0a08b -i td/osd-recovery-stats/3/new.json 2024-04-25T08:34:09.512 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:34:09.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:34:09.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAFSpmouhGNhAAXLCbaVb84EJrgG5AG3HlyA== --osd-uuid 9ff353be-724c-4f0e-a6f1-0dd810c0a08b 2024-04-25T08:34:09.565 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:09.566+0000 7fc9a1aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:09.568 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:09.570+0000 7fc9a1aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:09.569 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:09.570+0000 7fc9a1aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:09.569 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:09.570+0000 7fc9a1aff7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:34:09.569 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:09.570+0000 7fc9a1aff7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:34:09.569 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:09.570+0000 7fc9a1aff7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:34:09.629 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:09.630+0000 7fc9a1aff7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:34:18.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:34:18.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:34:18.165 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T08:34:18.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:34:18.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:34:18.739 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T08:34:18.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:34:18.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:34:18.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:34:18.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:34:18.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:34:18.765 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:18.766+0000 7fd0fcbae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:18.767 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:18.766+0000 7fd0fcbae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:18.768 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:18.770+0000 7fd0fcbae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:19.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:34:19.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:34:19.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:34:19.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:34:19.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:34:19.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:19.138 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:34:19.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:34:19.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:19.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:19.325 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:19.326+0000 7fd0fcbae7c0 -1 Falling back to public interface 2024-04-25T08:34:19.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:20.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:20.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:20.539 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:34:20.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:34:20.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:20.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:20.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:21.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:21.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:21.967 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:34:21.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:34:21.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:21.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:22.278 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:22.278+0000 7fd0fcbae7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:34:22.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:23.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:23.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:23.573 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:34:23.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:34:23.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:23.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:24.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:25.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:25.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:25.443 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:34:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:34:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:25.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:25.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:26.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:26.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:26.936 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:34:26.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:34:26.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:26.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:27.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:28.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:28.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:28.630 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:34:28.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:34:28.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:28.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:29.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:30.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:30.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:30.470 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:34:30.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:34:30.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:30.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:30.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:31.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:31.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:31.992 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:34:31.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:34:31.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:31.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:32.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:33.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:33.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:33.562 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:34:33.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:34:33.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:33.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:34.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:35.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:35.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:35.020 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:34:35.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:34:35.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:35.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:35.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:36.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:36.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:36.644 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:34:36.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:34:36.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:36.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:37.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:38.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:38.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:38.110 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:34:38.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:34:38.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:38.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:38.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:39.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:39.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:39.834 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:34:39.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:34:39.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:39.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:40.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:41.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:41.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:41.512 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:34:41.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:34:41.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:41.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:42.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:43.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:43.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:43.022 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:34:43.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:34:43.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:43.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:44.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:45.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:45.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:45.104 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:34:45.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:34:45.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:45.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:45.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:46.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:46.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:46.949 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:34:46.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:34:46.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:46.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:48.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:49.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:49.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:49.140 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:34:49.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:34:49.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:49.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:50.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:51.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:51.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:51.069 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:34:51.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:34:51.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:51.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:51.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:52.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:52.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:52.947 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:34:52.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:34:52.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:52.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:54.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:54.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:54.630 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:34:54.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:34:54.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:54.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:55.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:56.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:56.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:56.775 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T08:34:56.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:34:56.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:56.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:34:57.219 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/868225363,v1:127.0.0.1:6827/868225363] [v2:127.0.0.1:6828/868225363,v1:127.0.0.1:6829/868225363] exists,up 9ff353be-724c-4f0e-a6f1-0dd810c0a08b 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:34:57.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:34:57.224 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:34:57.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:34:57.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=019d304b-5ce8-4182-9207-062c2d1a489e 2024-04-25T08:34:57.226 INFO:tasks.workunit.client.0.smithi088.stdout:add osd4 019d304b-5ce8-4182-9207-062c2d1a489e 2024-04-25T08:34:57.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 019d304b-5ce8-4182-9207-062c2d1a489e' 2024-04-25T08:34:57.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:34:57.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxFSpmm4ETDxAAbcFte+V2vclv5DBSWujy5w== 2024-04-25T08:34:57.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxFSpmm4ETDxAAbcFte+V2vclv5DBSWujy5w=="}' 2024-04-25T08:34:57.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 019d304b-5ce8-4182-9207-062c2d1a489e -i td/osd-recovery-stats/4/new.json 2024-04-25T08:34:57.997 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:34:58.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:34:58.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxFSpmm4ETDxAAbcFte+V2vclv5DBSWujy5w== --osd-uuid 019d304b-5ce8-4182-9207-062c2d1a489e 2024-04-25T08:34:58.042 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:58.045+0000 7f95990367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:58.044 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:58.049+0000 7f95990367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:58.045 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:58.049+0000 7f95990367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:58.046 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:58.049+0000 7f95990367c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:34:58.046 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:58.049+0000 7f95990367c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:34:58.046 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:58.049+0000 7f95990367c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:34:58.167 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:34:58.173+0000 7f95990367c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:35:06.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:35:06.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:35:06.328 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd4 key to auth repository 2024-04-25T08:35:06.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:35:06.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:35:06.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:35:06.866 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.4 2024-04-25T08:35:06.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:35:06.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:35:06.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:35:06.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:35:06.898 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:06.901+0000 7fef4efc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:06.900 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:06.905+0000 7fef4efc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:06.901 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:06.905+0000 7fef4efc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:07.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:35:07.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:35:07.337 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:35:07.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:35:07.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:35:07.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:35:07.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:07.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:35:07.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:07.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:07.452 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:07.457+0000 7fef4efc97c0 -1 Falling back to public interface 2024-04-25T08:35:07.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:08.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:08.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:08.842 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:35:08.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:35:08.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:08.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:09.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:10.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:10.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:10.274 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:35:10.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:35:10.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:10.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:10.484 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:10.489+0000 7fef4efc97c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:35:10.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:11.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:11.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:11.811 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:35:11.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:35:11.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:11.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:12.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:13.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:13.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:13.450 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:35:13.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:35:13.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:13.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:14.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:15.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:15.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:15.199 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:35:15.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:35:15.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:15.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:15.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:16.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:16.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:16.881 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:35:16.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:35:16.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:16.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:17.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:18.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:18.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:18.291 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:35:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:35:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:18.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:19.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:19.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:19.972 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:35:19.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:35:19.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:19.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:20.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:21.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:21.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:21.559 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:35:21.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:35:21.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:21.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:23.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:23.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:23.372 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:35:23.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:35:23.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:23.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:24.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:25.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:25.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:25.072 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:35:25.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:35:25.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:25.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:25.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:26.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:26.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:26.523 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:35:26.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:35:26.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:26.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:27.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:28.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:28.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:28.081 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:35:28.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:35:28.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:28.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:28.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:29.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:29.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:29.599 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:35:29.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:35:29.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:29.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:30.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:31.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:31.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:31.336 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:35:31.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:35:31.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:31.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:32.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:33.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:33.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:33.444 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:35:33.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:35:33.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:33.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:34.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:35.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:35.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:35.460 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:35:35.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:35:35.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:35.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:36.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:37.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:37.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:37.504 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:35:37.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:35:37.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:37.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:38.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:39.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:39.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:39.523 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:35:39.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:35:39.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:39.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:40.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:41.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:41.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:41.640 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:35:41.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:35:41.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:41.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:42.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:43.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:43.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:43.883 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:35:43.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:35:43.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:43.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:35:44.336 INFO:tasks.workunit.client.0.smithi088.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3391391041,v1:127.0.0.1:6835/3391391041] [v2:127.0.0.1:6836/3391391041,v1:127.0.0.1:6837/3391391041] exists,up 019d304b-5ce8-4182-9207-062c2d1a489e 2024-04-25T08:35:44.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:35:44.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:35:44.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:35:44.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2024-04-25T08:35:44.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:35:44.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:35:44.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:35:44.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:35:44.341 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:35:44.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:35:44.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:35:44.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:35:44.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:35:44.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:35:44.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c50e315-309e-4ecd-8850-0d679eed4f80 2024-04-25T08:35:44.344 INFO:tasks.workunit.client.0.smithi088.stdout:add osd5 9c50e315-309e-4ecd-8850-0d679eed4f80 2024-04-25T08:35:44.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 9c50e315-309e-4ecd-8850-0d679eed4f80' 2024-04-25T08:35:44.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:35:44.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDgFSpmHYgMFhAA0sQ1tAdslf7Ne72QxZA1jg== 2024-04-25T08:35:44.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDgFSpmHYgMFhAA0sQ1tAdslf7Ne72QxZA1jg=="}' 2024-04-25T08:35:44.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c50e315-309e-4ecd-8850-0d679eed4f80 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:35:45.048 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:35:45.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:35:45.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgFSpmHYgMFhAA0sQ1tAdslf7Ne72QxZA1jg== --osd-uuid 9c50e315-309e-4ecd-8850-0d679eed4f80 2024-04-25T08:35:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:45.104+0000 7fbdae8487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:45.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:45.168+0000 7fbdae8487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:45.167 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:45.172+0000 7fbdae8487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:45.168 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:45.172+0000 7fbdae8487c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:35:45.168 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:45.172+0000 7fbdae8487c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:35:45.168 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:45.172+0000 7fbdae8487c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:35:45.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:45.216+0000 7fbdae8487c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:35:53.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:35:53.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:35:53.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:35:53.011 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd5 key to auth repository 2024-04-25T08:35:53.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:35:53.664 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.5 2024-04-25T08:35:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:35:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:35:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:35:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:35:53.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:35:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:53.700+0000 7f28278597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:53.728 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:53.732+0000 7f28278597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:53.729 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:53.732+0000 7f28278597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:54.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:35:54.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:35:54.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:35:54.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:35:54.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:35:54.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:54.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:35:54.126 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:35:54.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:54.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:35:54.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:54.316+0000 7f28278597c0 -1 Falling back to public interface 2024-04-25T08:35:54.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:55.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:55.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:55.552 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:35:55.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:35:55.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:55.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:35:55.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:56.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:56.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:56.959 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:35:56.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:35:56.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:56.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:35:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:35:57.084+0000 7f28278597c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:35:57.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:58.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:58.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:58.412 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:35:58.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:35:58.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:58.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:35:59.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:00.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:00.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:00.068 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:36:00.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:36:00.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:00.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:00.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:01.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:01.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:01.609 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:36:01.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:36:01.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:01.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:02.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:03.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:03.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:03.073 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:36:03.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:36:03.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:03.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:03.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:04.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:04.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:04.551 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:36:04.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:36:04.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:04.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:04.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:05.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:05.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:05.981 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:36:05.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:36:05.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:05.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:06.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:07.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:07.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:07.402 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:36:07.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:36:07.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:07.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:07.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:08.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:08.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:08.893 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:36:08.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:36:08.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:08.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:09.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:10.395 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:36:10.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:36:10.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:10.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:10.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:11.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:11.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:11.952 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:36:11.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:36:11.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:11.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:12.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:13.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:13.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:13.621 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:36:13.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:36:13.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:13.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:14.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:15.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:15.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:15.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:36:15.248 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:36:15.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:15.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:16.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:17.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:17.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:17.163 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:36:17.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:36:17.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:17.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:18.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:19.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:19.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:19.139 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:36:19.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:36:19.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:19.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:20.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:36:21.172 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:36:21.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:21.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:22.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:23.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:23.183 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:36:23.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:36:23.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:23.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:24.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:25.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:25.231 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:36:25.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:36:25.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:25.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:26.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:27.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:27.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:27.468 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T08:36:27.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:36:27.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:27.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:28.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:29.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:29.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:29.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:36:29.240 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T08:36:29.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:29.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:36:29.813 INFO:tasks.workunit.client.0.smithi088.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1906987699,v1:127.0.0.1:6843/1906987699] [v2:127.0.0.1:6844/1906987699,v1:127.0.0.1:6845/1906987699] exists,up 9c50e315-309e-4ecd-8850-0d679eed4f80 2024-04-25T08:36:29.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:36:29.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:36:29.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:36:29.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2024-04-25T08:36:29.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:36:30.590 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T08:36:30.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:36:31.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2024-04-25T08:36:32.857 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 4 2024-04-25T08:36:32.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2024-04-25T08:36:32.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:36:32.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:36:32.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:36:32.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:36:32.883 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:36:32.883 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:36:32.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:36:32.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:36:32.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:36:33.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:36:33.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:36:33.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:36:33.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:36:33.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:36:33.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:36:33.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:36:33.632 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:36:33.632 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:36:33.632 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:36:33.633 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:36:33.633 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:36:33.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:36:33.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:33.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:36:33.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836536 2024-04-25T08:36:33.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836536 2024-04-25T08:36:33.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836536' 2024-04-25T08:36:33.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:33.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:36:34.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673010 2024-04-25T08:36:34.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673010 2024-04-25T08:36:34.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836536 1-42949673010' 2024-04-25T08:36:34.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:34.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:36:34.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509479 2024-04-25T08:36:34.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509479 2024-04-25T08:36:34.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836536 1-42949673010 2-64424509479' 2024-04-25T08:36:34.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:34.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:36:34.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345950 2024-04-25T08:36:34.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345950 2024-04-25T08:36:34.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836536 1-42949673010 2-64424509479 3-85899345950' 2024-04-25T08:36:34.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:34.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:36:34.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182418 2024-04-25T08:36:34.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182418 2024-04-25T08:36:34.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836536 1-42949673010 2-64424509479 3-85899345950 4-107374182418' 2024-04-25T08:36:34.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:34.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:36:34.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018890 2024-04-25T08:36:34.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018890 2024-04-25T08:36:34.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836536 1-42949673010 2-64424509479 3-85899345950 4-107374182418 5-128849018890' 2024-04-25T08:36:34.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:34.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836536 2024-04-25T08:36:34.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:34.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:36:34.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836536 2024-04-25T08:36:34.742 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836536 2024-04-25T08:36:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-04-25T08:36:34.744 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836536 2024-04-25T08:36:34.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:36:35.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836534 -lt 21474836536 2024-04-25T08:36:35.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:36:36.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:36:36.166 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:36:36.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836536 -lt 21474836536 2024-04-25T08:36:36.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:36.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673010 2024-04-25T08:36:36.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:36.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:36:36.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673010 2024-04-25T08:36:36.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:36.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673010 2024-04-25T08:36:36.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2024-04-25T08:36:36.574 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673010 2024-04-25T08:36:36.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:36:36.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673010 -lt 42949673010 2024-04-25T08:36:36.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:36.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509479 2024-04-25T08:36:37.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:37.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:36:37.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509479 2024-04-25T08:36:37.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:37.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509479 2024-04-25T08:36:37.003 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509479 2024-04-25T08:36:37.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-04-25T08:36:37.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:36:37.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509480 -lt 64424509479 2024-04-25T08:36:37.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:37.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:37.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345950 2024-04-25T08:36:37.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:36:37.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345950 2024-04-25T08:36:37.425 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:37.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345950 2024-04-25T08:36:37.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345950' 2024-04-25T08:36:37.427 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345950 2024-04-25T08:36:37.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:36:37.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345951 -lt 85899345950 2024-04-25T08:36:37.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:37.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182418 2024-04-25T08:36:37.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:37.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:36:37.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182418 2024-04-25T08:36:37.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:37.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182418 2024-04-25T08:36:37.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182418' 2024-04-25T08:36:37.910 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182418 2024-04-25T08:36:37.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:36:38.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182418 -lt 107374182418 2024-04-25T08:36:38.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:38.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:38.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018890 2024-04-25T08:36:38.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:36:38.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018890 2024-04-25T08:36:38.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:38.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018890 2024-04-25T08:36:38.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2024-04-25T08:36:38.322 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 128849018890 2024-04-25T08:36:38.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:36:38.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018891 -lt 128849018890 2024-04-25T08:36:38.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:36:38.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:38.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:39.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:36:39.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:36:39.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:36:39.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:36:39.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:36:39.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:36:39.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:36:39.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:36:39.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:36:39.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:36:39.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:39.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:40.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:36:40.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:36:40.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:36:40.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2024-04-25T08:36:40.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:40.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2024-04-25T08:36:40.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:40.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2024-04-25T08:36:40.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:40.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2024-04-25T08:36:40.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:40.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2024-04-25T08:36:41.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:41.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2024-04-25T08:36:41.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:41.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2024-04-25T08:36:41.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:41.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2024-04-25T08:36:41.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:41.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2024-04-25T08:36:41.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:41.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2024-04-25T08:36:42.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:42.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2024-04-25T08:36:42.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:42.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2024-04-25T08:36:42.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:42.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2024-04-25T08:36:42.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:42.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2024-04-25T08:36:42.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:42.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2024-04-25T08:36:42.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:42.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2024-04-25T08:36:43.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:43.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2024-04-25T08:36:43.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:43.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2024-04-25T08:36:43.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:43.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2024-04-25T08:36:43.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:43.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2024-04-25T08:36:43.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:43.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2024-04-25T08:36:43.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:43.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2024-04-25T08:36:44.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:44.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2024-04-25T08:36:44.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:44.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2024-04-25T08:36:44.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:44.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2024-04-25T08:36:44.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:44.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2024-04-25T08:36:44.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:44.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2024-04-25T08:36:45.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:45.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2024-04-25T08:36:45.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:45.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2024-04-25T08:36:45.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:45.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2024-04-25T08:36:45.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:45.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2024-04-25T08:36:45.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:45.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2024-04-25T08:36:46.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:46.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2024-04-25T08:36:46.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:46.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2024-04-25T08:36:46.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:46.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2024-04-25T08:36:46.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:46.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2024-04-25T08:36:46.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:46.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2024-04-25T08:36:46.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:46.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2024-04-25T08:36:47.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:47.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2024-04-25T08:36:47.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:47.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2024-04-25T08:36:47.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:47.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2024-04-25T08:36:47.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:47.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2024-04-25T08:36:47.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:47.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2024-04-25T08:36:48.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:48.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2024-04-25T08:36:48.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:48.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2024-04-25T08:36:48.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:48.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2024-04-25T08:36:48.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:48.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2024-04-25T08:36:48.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:48.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2024-04-25T08:36:49.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:49.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2024-04-25T08:36:49.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:49.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2024-04-25T08:36:49.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:49.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2024-04-25T08:36:49.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:49.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2024-04-25T08:36:49.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:49.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2024-04-25T08:36:50.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:50.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2024-04-25T08:36:50.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:50.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2024-04-25T08:36:50.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:50.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2024-04-25T08:36:50.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:50.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2024-04-25T08:36:50.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:50.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2024-04-25T08:36:50.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:50.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2024-04-25T08:36:51.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:51.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2024-04-25T08:36:51.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:51.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2024-04-25T08:36:51.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:51.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2024-04-25T08:36:51.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:51.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2024-04-25T08:36:51.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:51.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2024-04-25T08:36:52.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:52.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2024-04-25T08:36:52.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:52.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2024-04-25T08:36:52.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:52.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2024-04-25T08:36:52.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:52.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2024-04-25T08:36:52.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:52.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2024-04-25T08:36:53.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:53.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2024-04-25T08:36:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:53.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2024-04-25T08:36:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2024-04-25T08:36:53.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2024-04-25T08:36:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2024-04-25T08:36:54.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:54.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2024-04-25T08:36:54.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:54.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2024-04-25T08:36:54.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:54.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2024-04-25T08:36:54.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:54.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2024-04-25T08:36:54.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:54.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2024-04-25T08:36:54.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:54.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2024-04-25T08:36:55.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:55.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2024-04-25T08:36:55.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:55.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2024-04-25T08:36:55.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:55.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2024-04-25T08:36:55.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:55.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2024-04-25T08:36:55.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:55.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2024-04-25T08:36:56.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:56.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2024-04-25T08:36:56.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:56.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2024-04-25T08:36:56.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:56.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2024-04-25T08:36:56.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:56.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2024-04-25T08:36:56.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:56.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2024-04-25T08:36:57.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:57.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2024-04-25T08:36:57.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:57.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2024-04-25T08:36:57.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:57.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2024-04-25T08:36:57.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:57.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2024-04-25T08:36:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:57.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2024-04-25T08:36:57.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:57.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2024-04-25T08:36:58.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:58.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2024-04-25T08:36:58.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:58.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2024-04-25T08:36:58.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:58.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2024-04-25T08:36:58.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:58.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2024-04-25T08:36:58.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:58.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2024-04-25T08:36:59.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:59.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2024-04-25T08:36:59.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:59.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2024-04-25T08:36:59.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:59.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2024-04-25T08:36:59.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:59.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2024-04-25T08:36:59.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:36:59.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2024-04-25T08:37:00.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:00.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2024-04-25T08:37:00.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:00.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2024-04-25T08:37:00.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:00.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2024-04-25T08:37:01.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:01.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2024-04-25T08:37:01.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:01.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2024-04-25T08:37:01.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:01.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2024-04-25T08:37:01.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:01.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2024-04-25T08:37:01.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:01.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2024-04-25T08:37:02.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:02.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2024-04-25T08:37:02.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:02.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2024-04-25T08:37:02.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:02.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2024-04-25T08:37:02.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:02.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2024-04-25T08:37:02.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:02.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2024-04-25T08:37:02.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:02.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2024-04-25T08:37:03.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:03.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2024-04-25T08:37:03.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:03.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2024-04-25T08:37:03.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:03.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2024-04-25T08:37:03.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:03.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2024-04-25T08:37:03.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:03.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2024-04-25T08:37:04.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:04.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2024-04-25T08:37:04.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:04.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2024-04-25T08:37:04.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:04.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2024-04-25T08:37:04.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:04.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2024-04-25T08:37:04.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:04.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2024-04-25T08:37:05.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:05.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2024-04-25T08:37:05.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:05.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2024-04-25T08:37:05.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:05.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2024-04-25T08:37:05.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:05.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2024-04-25T08:37:05.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:05.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2024-04-25T08:37:06.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:06.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2024-04-25T08:37:06.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:06.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2024-04-25T08:37:06.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:06.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2024-04-25T08:37:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:06.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2024-04-25T08:37:06.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:06.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2024-04-25T08:37:06.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:06.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2024-04-25T08:37:07.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:07.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2024-04-25T08:37:07.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:07.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2024-04-25T08:37:07.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:07.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2024-04-25T08:37:07.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:07.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2024-04-25T08:37:07.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:07.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2024-04-25T08:37:08.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:08.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2024-04-25T08:37:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2024-04-25T08:37:08.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:08.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2024-04-25T08:37:08.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:08.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2024-04-25T08:37:08.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:08.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2024-04-25T08:37:09.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:09.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2024-04-25T08:37:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:09.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2024-04-25T08:37:09.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:09.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2024-04-25T08:37:09.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:09.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2024-04-25T08:37:09.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:09.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2024-04-25T08:37:09.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:09.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2024-04-25T08:37:10.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:10.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2024-04-25T08:37:10.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:10.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2024-04-25T08:37:10.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:10.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2024-04-25T08:37:10.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:10.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2024-04-25T08:37:10.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:10.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2024-04-25T08:37:11.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:11.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2024-04-25T08:37:11.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:11.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2024-04-25T08:37:11.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:11.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2024-04-25T08:37:11.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:11.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2024-04-25T08:37:11.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:11.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2024-04-25T08:37:12.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2024-04-25T08:37:12.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:12.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2024-04-25T08:37:12.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2024-04-25T08:37:12.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:12.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2024-04-25T08:37:12.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:12.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2024-04-25T08:37:12.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:12.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2024-04-25T08:37:13.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:13.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2024-04-25T08:37:13.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:13.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2024-04-25T08:37:13.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:13.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2024-04-25T08:37:13.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:13.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2024-04-25T08:37:13.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:13.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2024-04-25T08:37:14.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:14.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2024-04-25T08:37:14.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:14.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2024-04-25T08:37:14.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:14.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2024-04-25T08:37:14.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:14.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2024-04-25T08:37:14.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:14.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2024-04-25T08:37:14.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:14.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2024-04-25T08:37:15.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:15.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2024-04-25T08:37:15.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:15.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2024-04-25T08:37:15.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:15.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2024-04-25T08:37:15.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:15.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2024-04-25T08:37:15.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:15.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2024-04-25T08:37:16.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:16.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2024-04-25T08:37:16.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:16.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2024-04-25T08:37:16.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:16.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2024-04-25T08:37:16.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:16.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2024-04-25T08:37:16.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:16.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2024-04-25T08:37:16.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:16.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2024-04-25T08:37:17.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:17.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2024-04-25T08:37:17.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:17.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2024-04-25T08:37:17.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:17.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2024-04-25T08:37:17.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:17.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2024-04-25T08:37:17.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:17.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2024-04-25T08:37:18.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:37:18.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2024-04-25T08:37:18.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:37:18.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:37:18.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:37:18.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:37:18.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:37:18.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2024-04-25T08:37:18.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2024-04-25T08:37:18.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:37:18.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:37:18.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:37:18.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:37:19.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2024-04-25T08:37:19.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2024-04-25T08:37:19.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-04-25T08:37:19.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj1 2024-04-25T08:37:19.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test obj1 2024-04-25T08:37:19.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:37:19.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-25T08:37:19.651 INFO:tasks.workunit.client.0.smithi088.stderr:0 2024-04-25T08:37:19.651 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:37:19.651 INFO:tasks.workunit.client.0.smithi088.stderr:4' 2024-04-25T08:37:19.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 4 2024-04-25T08:37:19.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2024-04-25T08:37:19.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2024-04-25T08:37:20.796 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is set 2024-04-25T08:37:20.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:37:20.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2024-04-25T08:37:21.481 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already out. 2024-04-25T08:37:21.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:37:21.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2024-04-25T08:37:22.369 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T08:37:22.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:37:22.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2024-04-25T08:37:24.432 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already out. 2024-04-25T08:37:24.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:37:24.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2024-04-25T08:37:25.729 INFO:tasks.workunit.client.0.smithi088.stderr:osd.4 is already out. 2024-04-25T08:37:25.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2024-04-25T08:37:27.049 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 2 2024-04-25T08:37:27.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2024-04-25T08:37:29.275 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is unset 2024-04-25T08:37:29.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:37:29.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:37:29.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:37:29.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:37:29.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:37:30.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2024-04-25T08:37:30.249 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:37:30.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2024-04-25T08:37:32.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2024-04-25T08:37:32.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:37:32.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:37:32.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:37:32.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:37:32.268 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:37:32.268 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:37:32.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:37:32.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:37:32.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:37:32.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:37:32.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:37:32.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:37:32.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:37:32.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:37:32.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:37:32.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:37:32.848 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:37:32.848 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:37:32.848 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:37:32.848 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:37:32.848 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:37:32.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:37:32.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:32.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:37:33.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836550 2024-04-25T08:37:33.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836550 2024-04-25T08:37:33.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550' 2024-04-25T08:37:33.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:33.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:37:33.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673024 2024-04-25T08:37:33.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673024 2024-04-25T08:37:33.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-42949673024' 2024-04-25T08:37:33.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:33.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:37:33.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509493 2024-04-25T08:37:33.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509493 2024-04-25T08:37:33.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-42949673024 2-64424509493' 2024-04-25T08:37:33.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:33.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:37:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345964 2024-04-25T08:37:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345964 2024-04-25T08:37:33.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-42949673024 2-64424509493 3-85899345964' 2024-04-25T08:37:33.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:33.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:37:33.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182431 2024-04-25T08:37:33.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182431 2024-04-25T08:37:33.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-42949673024 2-64424509493 3-85899345964 4-107374182431' 2024-04-25T08:37:33.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:37:33.694 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:37:33.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018904 2024-04-25T08:37:33.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018904 2024-04-25T08:37:33.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836550 1-42949673024 2-64424509493 3-85899345964 4-107374182431 5-128849018904' 2024-04-25T08:37:33.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:33.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836550 2024-04-25T08:37:33.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:33.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:37:33.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836550 2024-04-25T08:37:33.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:33.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836550 2024-04-25T08:37:33.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-04-25T08:37:33.850 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836550 2024-04-25T08:37:33.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:37:34.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836550 -lt 21474836550 2024-04-25T08:37:34.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:34.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673024 2024-04-25T08:37:34.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:34.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:37:34.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673024 2024-04-25T08:37:34.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:34.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673024 2024-04-25T08:37:34.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673024' 2024-04-25T08:37:34.409 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673024 2024-04-25T08:37:34.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:37:35.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673024 -lt 42949673024 2024-04-25T08:37:35.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:35.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509493 2024-04-25T08:37:35.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:35.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:37:35.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509493 2024-04-25T08:37:35.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:35.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509493 2024-04-25T08:37:35.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-04-25T08:37:35.112 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509493 2024-04-25T08:37:35.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:37:35.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509493 -lt 64424509493 2024-04-25T08:37:35.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:35.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345964 2024-04-25T08:37:35.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:35.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:37:35.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345964 2024-04-25T08:37:35.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:35.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345964 2024-04-25T08:37:35.763 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345964 2024-04-25T08:37:35.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345964' 2024-04-25T08:37:35.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:37:36.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345964 -lt 85899345964 2024-04-25T08:37:36.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:36.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:36.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182431 2024-04-25T08:37:36.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:37:36.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182431 2024-04-25T08:37:36.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:36.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182431 2024-04-25T08:37:36.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2024-04-25T08:37:36.215 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182431 2024-04-25T08:37:36.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:37:36.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182432 -lt 107374182431 2024-04-25T08:37:36.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:37:36.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018904 2024-04-25T08:37:36.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:37:36.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:37:36.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018904 2024-04-25T08:37:36.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:37:36.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018904 2024-04-25T08:37:36.889 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 128849018904 2024-04-25T08:37:36.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018904' 2024-04-25T08:37:36.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:37:37.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018905 -lt 128849018904 2024-04-25T08:37:37.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:37:37.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:37.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:38.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:37:38.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:38.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:38.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:38.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:38.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:38.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:38.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:38.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:38.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:38.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:38.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:37:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:37:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:37:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:39.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:39.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:39.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:39.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:39.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:39.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:39.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:39.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:39.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:39.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:39.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:39.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:39.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:40.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:40.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:40.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:40.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:40.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:40.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:40.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:40.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:40.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:41.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:37:41.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:37:41.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:41.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:41.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:41.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:41.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:41.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:41.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:41.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:41.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:41.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:41.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:41.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:41.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:41.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:41.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:42.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:42.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:42.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:37:42.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:37:42.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:42.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:42.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:43.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:43.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:43.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:43.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:43.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:43.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:43.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:43.035 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:43.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:43.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:43.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:43.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:44.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:44.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:44.032 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:44.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:37:44.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:37:44.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:44.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:44.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:44.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:44.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:44.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:44.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:44.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:44.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:44.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:44.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:45.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:45.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:45.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:45.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:45.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:45.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:45.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:45.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:45.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:45.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:45.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:45.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:45.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:46.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:37:46.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:37:46.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:46.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:46.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:46.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:46.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:46.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:46.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:46.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:46.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:46.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:46.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:47.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:47.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:47.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:47.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:47.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:47.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:48.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:37:48.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:37:48.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:48.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:48.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:48.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:48.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:48.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:48.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:48.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:48.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:48.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:48.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:48.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:48.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:48.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:48.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:49.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:49.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:49.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:49.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:49.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:49.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:49.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:49.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:49.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:50.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:37:50.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:37:50.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:50.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:50.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:50.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:50.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:50.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:50.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:50.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:50.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:50.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:50.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:50.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:50.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:50.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:50.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:51.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:51.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:51.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:51.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:51.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:51.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:51.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:51.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:51.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:52.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:37:52.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:37:52.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:52.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:52.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:52.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:52.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:52.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:52.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:52.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:52.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:52.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:52.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:52.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:52.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:52.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:52.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:53.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:53.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:53.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:53.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:53.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:53.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:53.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:37:53.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:37:53.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:53.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:53.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:53.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:53.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:53.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:53.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:53.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:53.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:53.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:53.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:54.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:54.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:54.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:54.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:54.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:54.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:54.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:54.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:54.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:54.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:54.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:54.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:54.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:55.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:37:55.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:37:55.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:55.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:55.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:55.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:55.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:55.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:55.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:55.809 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:55.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:55.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:55.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:56.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:56.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:56.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:56.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:56.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:56.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:56.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:56.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:56.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:56.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:56.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:56.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:56.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:57.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:37:57.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:37:57.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:57.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:57.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:57.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:57.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:57.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:57.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:57.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:57.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:57.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:58.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:58.051 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:58.051 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:58.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:58.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:58.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:58.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:58.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:58.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:58.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:58.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:58.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:58.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:59.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:37:59.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:37:59.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:59.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:59.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:59.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:59.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:59.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:59.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:59.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:59.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:59.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:59.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:59.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:59.908 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:59.908 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:59.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:00.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:00.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:00.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:00.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:00.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:00.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:00.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:00.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:00.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:01.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:38:01.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:38:01.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:01.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:01.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:01.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:01.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:01.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:01.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:01.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:01.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:01.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:01.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:02.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:38:02.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:02.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:02.163 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:38:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:38:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:38:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:38:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:38:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:38:02.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:38:02.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:38:02.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:38:03.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:38:03.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:38:03.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:38:03.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:38:03.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:38:03.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:38:03.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:38:03.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:38:03.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:38:03.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:38:03.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:38:03.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:38:03.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:38:03.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:38:03.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:38:03.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:38:03.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:38:04.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:38:04.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:38:04.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:38:04.541 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:38:04.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:38:04.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:38:04.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:38:04.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:38:04.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2024-04-25T08:38:04.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2024-04-25T08:38:04.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2024-04-25T08:38:04.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:04.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:38:04.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:38:04.953 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:04.953 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:38:04.953 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:38:04.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2024-04-25T08:38:04.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:38:04.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:04.973 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:04.973 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2024-04-25T08:38:04.974 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:38:04.974 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:38:04.974 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:38:04.979 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:38:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:38:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:38:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:38:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:38:04.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:38:04.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:38:04.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:04.988 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:04.988 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2024-04-25T08:38:04.989 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:38:04.989 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:38:04.989 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:04.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-04-25T08:38:04.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-04-25T08:38:04.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T08:38:04.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:38:04.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:38:04.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:38:04.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:38:04.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:04.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:38:04.993 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:05.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:38:05.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:38:05.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:38:05.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:38:05.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:38:05.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:38:05.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:38:05.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:05.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2024-04-25T08:38:05.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:38:05.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2024-04-25T08:38:05.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:05.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2024-04-25T08:38:05.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2024-04-25T08:38:05.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:05.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2024-04-25T08:38:05.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2024-04-25T08:38:05.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:38:05.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:38:05.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:38:05.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:38:05.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:38:05.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2024-04-25T08:38:05.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:38:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2024-04-25T08:38:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2024-04-25T08:38:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:38:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:38:05.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:38:05.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:38:05.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2024-04-25T08:38:05.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:38:05.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:38:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T08:38:05.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2024-04-25T08:38:05.610 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:38:05.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:38:05.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:38:05.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:38:05.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:38:06.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:38:06.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:38:06.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:38:06.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:38:06.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:38:06.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:38:06.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:38:06.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:38:06.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:38:06.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:38:06.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:38:06.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:38:06.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:38:06.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:38:06.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:38:06.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:38:06.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:38:06.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:38:06.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:38:06.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:38:06.942 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:38:06.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:38:06.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:38:06.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:38:06.987 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:38:06.987 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:06.987 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:06.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:38:06.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:38:06.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:38:06.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:38:06.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:38:06.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:38:06.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:38:06.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:38:06.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:38:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:38:06.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:38:06.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:38:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:38:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:38:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:38:06.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:38:06.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:38:06.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:38:06.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:38:06.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:38:06.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:38:06.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:38:06.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:38:06.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:38:06.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:38:06.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:38:06.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:38:06.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:38:06.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:38:06.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:38:06.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:06.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:06.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:38:06.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:38:06.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:38:06.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:38:07.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:38:07.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:07.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:07.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274961 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:38:07.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:38:07.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:38:07.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:07.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:07.498 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:07.498 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:07.498 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:07.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:38:07.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:38:07.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:38:07.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:38:07.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:38:07.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:38:07.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:38:07.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:38:07.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:38:07.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:38:07.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:38:07.718 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:38:07.718 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:07.718 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:07.718 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:38:07.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:38:07.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get fsid 2024-04-25T08:38:07.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:38:07.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:38:07.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:38:07.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:38:07.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:38:07.834 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:38:07.835 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:38:07.835 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:38:07.835 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:38:07.835 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:07.835 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:07.836 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:38:07.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:38:07.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get mon_host 2024-04-25T08:38:07.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2024-04-25T08:38:07.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:38:07.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:38:07.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:38:07.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:38:07.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:38:07.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:38:08.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:38:08.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:08.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:08.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:08.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:08.876 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:08.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:38:08.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:38:08.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:38:08.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:38:08.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:38:08.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:38:08.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:38:08.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7271e4ea-9651-46eb-a3df-13fcc4a1f324 2024-04-25T08:38:08.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7271e4ea-9651-46eb-a3df-13fcc4a1f324' 2024-04-25T08:38:08.917 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 7271e4ea-9651-46eb-a3df-13fcc4a1f324 2024-04-25T08:38:08.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:38:08.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwFipm5dYQOBAAHS1pfwhus73prGm2HT1Cxg== 2024-04-25T08:38:08.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwFipm5dYQOBAAHS1pfwhus73prGm2HT1Cxg=="}' 2024-04-25T08:38:08.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7271e4ea-9651-46eb-a3df-13fcc4a1f324 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:38:09.269 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:38:09.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:38:09.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwFipm5dYQOBAAHS1pfwhus73prGm2HT1Cxg== --osd-uuid 7271e4ea-9651-46eb-a3df-13fcc4a1f324 2024-04-25T08:38:09.312 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:09.318+0000 7fc09477c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:09.354+0000 7fc09477c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:09.348 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:09.354+0000 7fc09477c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:09.349 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:09.354+0000 7fc09477c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:38:09.349 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:09.354+0000 7fc09477c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:38:09.349 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:09.354+0000 7fc09477c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:38:09.386 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:09.394+0000 7fc09477c7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:38:15.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:38:15.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:38:15.873 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:38:15.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:38:15.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:38:16.412 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:38:16.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:38:16.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:38:16.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:38:16.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:38:16.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:38:16.443 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:16.449+0000 7f332d7727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:16.445 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:16.449+0000 7f332d7727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:16.446 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:16.453+0000 7f332d7727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:16.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:38:16.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:38:16.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:38:16.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:38:16.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:38:16.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:16.854 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:38:16.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:38:16.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:16.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:17.004 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:17.009+0000 7f332d7727c0 -1 Falling back to public interface 2024-04-25T08:38:17.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:18.292 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:38:18.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:38:18.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:18.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:18.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:19.694 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:19.701+0000 7f332d7727c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:38:19.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:19.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:19.715 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:38:19.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:38:19.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:19.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:20.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:21.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:21.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:21.403 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:38:21.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:38:21.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:21.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:21.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:22.890 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:38:22.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:38:22.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:22.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:23.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:24.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:24.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:24.308 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:38:24.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:38:24.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:24.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:24.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:25.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:25.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:25.870 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:38:25.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:38:25.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:25.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:26.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:27.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:27.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:27.300 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:38:27.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:38:27.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:27.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:27.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:28.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:28.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:28.881 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:38:28.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:38:28.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:28.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:29.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:30.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:30.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:30.343 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:38:30.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:38:30.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:30.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:30.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:31.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:31.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:31.881 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:38:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:38:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:31.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:32.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:33.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:33.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:33.313 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:38:33.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:38:33.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:33.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:33.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:34.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:34.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:34.738 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:38:34.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:38:34.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:34.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:35.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:36.262 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:38:36.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:38:36.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:36.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:36.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:37.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:37.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:37.945 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:38:37.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:38:37.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:37.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:39.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:40.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:40.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:40.334 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:38:40.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:38:40.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:40.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:41.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:42.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:38:42.437 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:38:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:42.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:43.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:44.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:44.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:44.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:38:44.285 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:38:44.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:44.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:45.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:46.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:46.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:46.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:38:46.199 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:38:46.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:46.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:38:46.609 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1526650055,v1:127.0.0.1:6803/1526650055] [v2:127.0.0.1:6804/1526650055,v1:127.0.0.1:6805/1526650055] exists,up 7271e4ea-9651-46eb-a3df-13fcc4a1f324 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:38:46.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:38:46.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:38:46.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:38:46.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:38:46.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=41b1ab7f-9184-40e2-96f1-fb2022499776 2024-04-25T08:38:46.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 41b1ab7f-9184-40e2-96f1-fb2022499776' 2024-04-25T08:38:46.616 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 41b1ab7f-9184-40e2-96f1-fb2022499776 2024-04-25T08:38:46.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:38:46.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWFipm93sWJhAATDykc11or9JMM7+E3H/qVg== 2024-04-25T08:38:46.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWFipm93sWJhAATDykc11or9JMM7+E3H/qVg=="}' 2024-04-25T08:38:46.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 41b1ab7f-9184-40e2-96f1-fb2022499776 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:38:47.185 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:38:47.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:38:47.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWFipm93sWJhAATDykc11or9JMM7+E3H/qVg== --osd-uuid 41b1ab7f-9184-40e2-96f1-fb2022499776 2024-04-25T08:38:47.235 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:47.241+0000 7f53831b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:47.256 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:47.261+0000 7f53831b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:47.257 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:47.261+0000 7f53831b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:47.258 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:47.265+0000 7f53831b17c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:38:47.258 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:47.265+0000 7f53831b17c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:38:47.258 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:47.265+0000 7f53831b17c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:38:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:47.301+0000 7f53831b17c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:38:54.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:38:54.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:38:54.200 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:38:54.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:38:54.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:38:54.756 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:38:54.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:38:54.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:38:54.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:38:54.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:38:54.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:38:54.787 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:54.793+0000 7fb2c00417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:54.811 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:54.817+0000 7fb2c00417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:54.813 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:54.817+0000 7fb2c00417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:55.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:38:55.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:38:55.201 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:38:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:38:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:38:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:38:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:38:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:55.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:55.387 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:55.393+0000 7fb2c00417c0 -1 Falling back to public interface 2024-04-25T08:38:55.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:56.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:56.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:56.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:38:56.635 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:38:56.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:56.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:57.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:58.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:58.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:58.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:38:58.039 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:38:58.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:58.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:58.197 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:38:58.205+0000 7fb2c00417c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:38:58.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:59.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:59.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:59.563 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:38:59.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:38:59.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:59.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:00.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:01.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:01.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:01.069 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:39:01.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:39:01.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:01.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:01.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:02.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:02.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:02.514 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:39:02.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:39:02.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:02.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:03.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:04.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:04.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:04.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:39:04.081 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:39:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:04.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:04.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:05.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:05.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:05.731 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:39:05.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:39:05.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:05.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:06.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:07.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:07.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:07.255 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:39:07.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:39:07.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:07.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:07.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:08.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:08.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:08.662 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:39:08.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:39:08.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:08.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:09.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:10.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:10.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:10.224 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:39:10.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:39:10.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:10.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:10.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:11.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:11.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:11.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:39:11.759 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:39:11.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:11.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:12.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:13.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:13.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:13.280 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:39:13.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:39:13.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:13.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:13.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:14.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:14.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:14.724 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:39:14.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:39:14.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:14.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:15.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:16.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:16.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:16.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:39:16.510 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:39:16.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:16.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:17.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:18.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:18.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:18.290 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:39:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:39:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:19.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:20.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:20.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:20.385 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:39:20.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:39:20.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:20.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:21.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:22.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:22.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:22.115 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:39:22.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:39:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:22.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:23.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:24.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:24.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:24.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:39:24.225 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:39:24.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:24.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:39:24.839 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1335489746,v1:127.0.0.1:6811/1335489746] [v2:127.0.0.1:6812/1335489746,v1:127.0.0.1:6813/1335489746] exists,up 41b1ab7f-9184-40e2-96f1-fb2022499776 2024-04-25T08:39:24.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:39:24.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:39:24.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:39:24.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:39:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:39:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:39:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:39:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:39:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:39:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:39:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:39:24.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:39:24.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:39:24.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00ea7939-bae2-4754-9100-de60648a79ad 2024-04-25T08:39:24.847 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 00ea7939-bae2-4754-9100-de60648a79ad 2024-04-25T08:39:24.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 00ea7939-bae2-4754-9100-de60648a79ad' 2024-04-25T08:39:24.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:39:24.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8FipmfGYxNBAAQKu0PfUD1etlYQeuFE4/qQ== 2024-04-25T08:39:24.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8FipmfGYxNBAAQKu0PfUD1etlYQeuFE4/qQ=="}' 2024-04-25T08:39:24.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00ea7939-bae2-4754-9100-de60648a79ad -i td/osd-recovery-stats/2/new.json 2024-04-25T08:39:25.492 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:39:25.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:39:25.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8FipmfGYxNBAAQKu0PfUD1etlYQeuFE4/qQ== --osd-uuid 00ea7939-bae2-4754-9100-de60648a79ad 2024-04-25T08:39:25.539 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:25.544+0000 7f25f2b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:25.580 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:25.584+0000 7f25f2b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:25.581 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:25.588+0000 7f25f2b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:25.582 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:25.588+0000 7f25f2b4d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:39:25.583 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:25.588+0000 7f25f2b4d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:39:25.583 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:25.588+0000 7f25f2b4d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:39:25.680 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:25.684+0000 7f25f2b4d7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:39:32.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:39:32.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:39:32.348 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:39:32.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:39:32.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:39:32.904 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:39:32.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:39:32.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:39:32.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:39:32.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:39:32.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:39:32.930 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:32.936+0000 7f536516c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:32.932 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:32.936+0000 7f536516c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:32.933 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:32.940+0000 7f536516c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:33.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:39:33.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:39:33.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:39:33.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:39:33.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:39:33.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:33.390 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:39:33.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:39:33.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:33.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:33.484 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:33.488+0000 7f536516c7c0 -1 Falling back to public interface 2024-04-25T08:39:33.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:34.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:34.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:34.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:39:34.860 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:39:34.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:34.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:35.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:36.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:36.267 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:39:36.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:39:36.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:36.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:36.347 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:39:36.352+0000 7f536516c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:39:36.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:37.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:37.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:37.723 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:39:37.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:39:37.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:37.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:38.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:39.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:39.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:39.281 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:39:39.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:39:39.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:39.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:39.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:40.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:40.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:40.747 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:39:40.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:39:40.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:40.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:41.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:42.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:39:42.317 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:39:42.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:42.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:42.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:43.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:43.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:43.870 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:39:43.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:39:43.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:43.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:44.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:45.439 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:39:45.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:39:45.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:45.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:45.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:46.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:46.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:46.881 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:39:46.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:39:46.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:46.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:47.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:48.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:48.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:48.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:39:48.422 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:39:48.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:48.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:48.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:49.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:49.869 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:39:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:39:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:49.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:50.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:51.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:51.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:51.417 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:39:51.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:39:51.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:51.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:52.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:53.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:53.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:53.682 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:39:53.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:39:53.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:53.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:54.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:55.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:55.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:55.374 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:39:55.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:39:55.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:55.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:56.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:57.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:57.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:57.356 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:39:57.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:39:57.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:57.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:58.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:59.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:59.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:59.114 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:39:59.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:39:59.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:59.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:59.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:00.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:00.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:00.882 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:40:00.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:40:00.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:00.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:01.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:02.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:02.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:02.676 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:40:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:40:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:02.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:03.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:04.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:04.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:04.159 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:40:04.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:40:04.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:04.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:40:04.621 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3201393179,v1:127.0.0.1:6819/3201393179] [v2:127.0.0.1:6820/3201393179,v1:127.0.0.1:6821/3201393179] exists,up 00ea7939-bae2-4754-9100-de60648a79ad 2024-04-25T08:40:04.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:40:04.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:40:04.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:40:04.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2024-04-25T08:40:04.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:40:04.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:40:04.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:40:04.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:40:04.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:40:04.626 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:40:04.626 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:40:04.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:40:04.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:40:04.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:40:04.628 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:40:04.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43d3bf00-68ce-498b-a905-16493e56ecce 2024-04-25T08:40:04.628 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 43d3bf00-68ce-498b-a905-16493e56ecce 2024-04-25T08:40:04.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 43d3bf00-68ce-498b-a905-16493e56ecce' 2024-04-25T08:40:04.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:40:04.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkFipmY99PJxAAJ+tYRLuInxAKh7MvG1iypw== 2024-04-25T08:40:04.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkFipmY99PJxAAJ+tYRLuInxAKh7MvG1iypw=="}' 2024-04-25T08:40:04.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43d3bf00-68ce-498b-a905-16493e56ecce -i td/osd-recovery-stats/3/new.json 2024-04-25T08:40:05.174 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:40:05.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:40:05.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkFipmY99PJxAAJ+tYRLuInxAKh7MvG1iypw== --osd-uuid 43d3bf00-68ce-498b-a905-16493e56ecce 2024-04-25T08:40:05.226 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:05.231+0000 7f837815d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:05.322 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:05.327+0000 7f837815d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:05.324 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:05.327+0000 7f837815d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:05.324 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:05.327+0000 7f837815d7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:40:05.324 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:05.327+0000 7f837815d7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:40:05.324 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:05.327+0000 7f837815d7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:40:05.367 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:05.371+0000 7f837815d7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:40:12.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:40:12.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:40:12.472 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T08:40:12.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:40:12.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:40:13.034 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T08:40:13.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:40:13.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:40:13.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:40:13.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:40:13.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:40:13.066 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:13.071+0000 7f85052d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:13.068 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:13.071+0000 7f85052d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:13.069 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:13.075+0000 7f85052d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:13.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:40:13.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:40:13.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:40:13.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:40:13.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:40:13.432 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:40:13.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:13.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:40:13.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:13.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:13.611 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:13.615+0000 7f85052d07c0 -1 Falling back to public interface 2024-04-25T08:40:13.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:14.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:14.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:14.851 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:40:14.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:40:14.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:14.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:15.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:16.265 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:16.271+0000 7f85052d07c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:40:16.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:16.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:16.305 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:40:16.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:40:16.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:16.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:17.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:18.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:18.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:18.127 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:40:18.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:40:18.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:18.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:18.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:19.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:19.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:19.636 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:40:19.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:40:19.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:19.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:20.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:21.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:21.279 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:40:21.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:40:21.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:21.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:21.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:22.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:22.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:22.767 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:40:22.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:40:22.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:22.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:23.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:24.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:24.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:24.423 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:40:24.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:40:24.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:24.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:24.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:25.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:25.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:25.881 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:40:25.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:40:25.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:25.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:26.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:27.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:27.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:27.357 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:40:27.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:40:27.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:27.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:27.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:28.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:28.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:28.886 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:40:28.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:40:28.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:28.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:29.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:30.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:30.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:30.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:40:30.384 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:40:30.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:30.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:30.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:31.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:31.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:31.819 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:40:31.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:40:31.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:31.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:32.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:33.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:33.678 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:40:33.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:40:33.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:33.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:34.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:35.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:35.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:35.393 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:40:35.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:40:35.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:35.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:36.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:37.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:37.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:37.172 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:40:37.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:40:37.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:37.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:37.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:38.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:38.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:38.698 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:40:38.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:40:38.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:38.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:39.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:40.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:40.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:40.712 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:40:40.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:40:40.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:40.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:41.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:42.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:42.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:42.699 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:40:42.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:40:42.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:42.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:43.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:44.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:44.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:44.251 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:40:44.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:40:44.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:44.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:40:44.691 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2095044535,v1:127.0.0.1:6827/2095044535] [v2:127.0.0.1:6828/2095044535,v1:127.0.0.1:6829/2095044535] exists,up 43d3bf00-68ce-498b-a905-16493e56ecce 2024-04-25T08:40:44.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:40:44.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:40:44.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:40:44.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:40:44.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0682f6d6-5917-44b6-8bbb-3af57f406f18 2024-04-25T08:40:44.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 0682f6d6-5917-44b6-8bbb-3af57f406f18' 2024-04-25T08:40:44.698 INFO:tasks.workunit.client.0.smithi088.stdout:add osd4 0682f6d6-5917-44b6-8bbb-3af57f406f18 2024-04-25T08:40:44.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:40:44.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMFypmVDFXKxAA7Z4DCyEVDVEtG8QVK1ORsg== 2024-04-25T08:40:44.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMFypmVDFXKxAA7Z4DCyEVDVEtG8QVK1ORsg=="}' 2024-04-25T08:40:44.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0682f6d6-5917-44b6-8bbb-3af57f406f18 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:40:45.245 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:40:45.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:40:45.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMFypmVDFXKxAA7Z4DCyEVDVEtG8QVK1ORsg== --osd-uuid 0682f6d6-5917-44b6-8bbb-3af57f406f18 2024-04-25T08:40:45.297 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:45.302+0000 7f6764dfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:45.330 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:45.334+0000 7f6764dfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:45.331 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:45.334+0000 7f6764dfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:45.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:45.334+0000 7f6764dfb7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:40:45.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:45.334+0000 7f6764dfb7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:40:45.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:45.334+0000 7f6764dfb7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:40:45.512 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:45.518+0000 7f6764dfb7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:40:52.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:40:52.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:40:52.251 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd4 key to auth repository 2024-04-25T08:40:52.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:40:52.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:40:52.814 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.4 2024-04-25T08:40:52.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:40:52.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:40:52.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:40:52.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:40:52.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:40:52.847 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:52.850+0000 7f18c653e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:52.870 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:52.874+0000 7f18c653e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:52.871 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:52.874+0000 7f18c653e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:53.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:40:53.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:40:53.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:40:53.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:40:53.225 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:40:53.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:40:53.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:53.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:40:53.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:53.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:53.426 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:53.430+0000 7f18c653e7c0 -1 Falling back to public interface 2024-04-25T08:40:53.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:54.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:54.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:54.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:40:54.637 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:40:54.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:54.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:56.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:56.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:56.040 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:40:56.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:40:56.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:56.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:56.183 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:40:56.186+0000 7f18c653e7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:40:56.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:57.557 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:40:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:40:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:57.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:58.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:58.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:58.995 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:40:58.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:40:58.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:58.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:59.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:00.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:00.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:00.598 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:41:00.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:41:00.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:00.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:01.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:02.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:02.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:02.049 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:41:02.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:41:02.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:02.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:02.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:03.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:03.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:03.708 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:41:03.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:41:03.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:03.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:04.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:05.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:05.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:05.242 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:41:05.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:41:05.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:05.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:05.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:06.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:06.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:06.747 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:41:06.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:41:06.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:06.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:07.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:08.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:08.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:08.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:41:08.371 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:41:08.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:08.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:08.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:09.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:09.819 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:41:09.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:09.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:41:09.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:09.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:11.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:11.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:11.397 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:41:11.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:41:11.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:11.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:12.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:13.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:13.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:13.082 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:41:13.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:41:13.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:13.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:13.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:14.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:14.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:14.936 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:41:14.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:41:14.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:14.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:15.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:16.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:16.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:16.807 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:41:16.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:41:16.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:17.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:18.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:18.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:18.455 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:41:18.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:41:18.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:18.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:19.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:20.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:20.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:20.191 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:41:20.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:41:20.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:20.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:21.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:22.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:22.153 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:41:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:41:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:22.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:23.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:24.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:24.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:24.045 INFO:tasks.workunit.client.0.smithi088.stdout:19 2024-04-25T08:41:24.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:41:24.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:24.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:41:24.512 INFO:tasks.workunit.client.0.smithi088.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/209022854,v1:127.0.0.1:6835/209022854] [v2:127.0.0.1:6836/209022854,v1:127.0.0.1:6837/209022854] exists,up 0682f6d6-5917-44b6-8bbb-3af57f406f18 2024-04-25T08:41:24.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:41:24.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:41:24.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:41:24.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:41:24.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:41:24.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:41:24.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:41:24.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:41:24.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:41:24.518 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:41:24.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=976dc1ff-bf41-49fc-9b14-d3500a5afcd4 2024-04-25T08:41:24.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 976dc1ff-bf41-49fc-9b14-d3500a5afcd4' 2024-04-25T08:41:24.519 INFO:tasks.workunit.client.0.smithi088.stdout:add osd5 976dc1ff-bf41-49fc-9b14-d3500a5afcd4 2024-04-25T08:41:24.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:41:24.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0Fypm+gd+IBAAgwlDSLrHXwvM2QGmlCypXA== 2024-04-25T08:41:24.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0Fypm+gd+IBAAgwlDSLrHXwvM2QGmlCypXA=="}' 2024-04-25T08:41:24.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 976dc1ff-bf41-49fc-9b14-d3500a5afcd4 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:41:25.019 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:41:25.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:41:25.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0Fypm+gd+IBAAgwlDSLrHXwvM2QGmlCypXA== --osd-uuid 976dc1ff-bf41-49fc-9b14-d3500a5afcd4 2024-04-25T08:41:25.065 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:25.074+0000 7f230b3297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:25.147 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:25.154+0000 7f230b3297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:25.149 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:25.158+0000 7f230b3297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:25.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:25.158+0000 7f230b3297c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:41:25.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:25.158+0000 7f230b3297c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:41:25.150 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:25.158+0000 7f230b3297c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:41:25.243 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:25.250+0000 7f230b3297c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:41:32.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:41:32.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:41:32.223 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd5 key to auth repository 2024-04-25T08:41:32.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:41:32.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:41:32.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:41:32.788 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.5 2024-04-25T08:41:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:41:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:41:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:41:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:41:32.813 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:32.822+0000 7fd638bc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:32.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:32.822+0000 7fd638bc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:32.816 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:32.826+0000 7fd638bc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:33.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:41:33.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:41:33.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:41:33.242 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:41:33.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:41:33.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:41:33.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:33.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:41:33.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:33.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:33.362 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:33.370+0000 7fd638bc17c0 -1 Falling back to public interface 2024-04-25T08:41:33.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:34.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:34.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:34.717 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:41:34.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:41:34.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:34.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:35.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:36.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:36.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:36.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:41:36.122 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:41:36.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:36.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:36.184 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:41:36.189+0000 7fd638bc17c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:41:36.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:37.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:37.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:37.568 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:41:37.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:41:37.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:37.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:38.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:39.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:41:39.118 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:41:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:39.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:39.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:40.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:40.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:40.584 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:41:40.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:41:40.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:40.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:41.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:42.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:42.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:42.129 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:41:42.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:41:42.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:42.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:42.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:43.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:43.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:43.577 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:41:43.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:41:43.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:43.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:44.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:45.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:45.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:45.098 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:41:45.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:41:45.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:45.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:45.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:46.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:46.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:46.542 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:41:46.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:41:46.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:46.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:47.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:48.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:48.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:48.107 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:41:48.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:41:48.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:48.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:48.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:49.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:49.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:49.546 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:41:49.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:41:49.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:49.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:50.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:51.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:51.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:51.130 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:41:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:41:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:51.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:52.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:52.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:52.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:41:52.988 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:41:52.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:52.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:53.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:54.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:54.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:54.783 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:41:54.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:41:54.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:54.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:55.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:56.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:56.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:56.798 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:41:56.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:41:56.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:56.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:57.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:58.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:58.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:58.584 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:41:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:41:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:58.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:59.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:00.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:00.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:00.412 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:42:00.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:42:00.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:00.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:01.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:02.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:02.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:02.163 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:42:02.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:42:02.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:02.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:42:02.633 INFO:tasks.workunit.client.0.smithi088.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2974980440,v1:127.0.0.1:6843/2974980440] [v2:127.0.0.1:6844/2974980440,v1:127.0.0.1:6845/2974980440] exists,up 976dc1ff-bf41-49fc-9b14-d3500a5afcd4 2024-04-25T08:42:02.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:42:02.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:42:02.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:42:02.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2024-04-25T08:42:02.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:42:03.382 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T08:42:03.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:42:04.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2024-04-25T08:42:05.488 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 2 2024-04-25T08:42:05.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2024-04-25T08:42:05.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:42:05.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:42:05.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:42:05.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:42:05.510 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:42:05.510 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:42:05.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:42:05.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:42:05.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:42:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:42:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:42:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:42:05.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:42:05.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:42:05.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:06.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:42:06.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836527 2024-04-25T08:42:06.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836527 2024-04-25T08:42:06.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527' 2024-04-25T08:42:06.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:06.393 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:42:06.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673001 2024-04-25T08:42:06.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673001 2024-04-25T08:42:06.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001' 2024-04-25T08:42:06.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:06.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:42:06.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509474 2024-04-25T08:42:06.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509474 2024-04-25T08:42:06.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509474' 2024-04-25T08:42:06.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:06.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345940 2024-04-25T08:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345940 2024-04-25T08:42:06.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509474 3-85899345940' 2024-04-25T08:42:06.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:06.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:42:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182414 2024-04-25T08:42:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182414 2024-04-25T08:42:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509474 3-85899345940 4-107374182414' 2024-04-25T08:42:06.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:06.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:42:07.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018886 2024-04-25T08:42:07.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018886 2024-04-25T08:42:07.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509474 3-85899345940 4-107374182414 5-128849018886' 2024-04-25T08:42:07.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:07.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836527 2024-04-25T08:42:07.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:07.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:42:07.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836527 2024-04-25T08:42:07.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:07.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836527 2024-04-25T08:42:07.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-04-25T08:42:07.143 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836527 2024-04-25T08:42:07.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:42:07.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836524 -lt 21474836527 2024-04-25T08:42:07.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:42:08.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:42:08.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:42:08.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836527 2024-04-25T08:42:08.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:08.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673001 2024-04-25T08:42:08.998 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:09.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:42:09.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673001 2024-04-25T08:42:09.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:09.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673001 2024-04-25T08:42:09.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-04-25T08:42:09.003 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673001 2024-04-25T08:42:09.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:42:09.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673001 -lt 42949673001 2024-04-25T08:42:09.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:09.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509474 2024-04-25T08:42:09.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:09.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:42:09.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509474 2024-04-25T08:42:09.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:09.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509474 2024-04-25T08:42:09.460 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509474 2024-04-25T08:42:09.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-04-25T08:42:09.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:42:09.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509475 -lt 64424509474 2024-04-25T08:42:09.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:09.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345940 2024-04-25T08:42:09.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:09.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:42:09.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345940 2024-04-25T08:42:09.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:09.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345940 2024-04-25T08:42:09.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345940' 2024-04-25T08:42:09.911 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345940 2024-04-25T08:42:09.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:42:10.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345941 -lt 85899345940 2024-04-25T08:42:10.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:10.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182414 2024-04-25T08:42:10.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:10.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:42:10.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182414 2024-04-25T08:42:10.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:10.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182414 2024-04-25T08:42:10.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-04-25T08:42:10.322 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182414 2024-04-25T08:42:10.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:42:10.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182415 -lt 107374182414 2024-04-25T08:42:10.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:10.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018886 2024-04-25T08:42:10.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:10.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:42:10.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018886 2024-04-25T08:42:10.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:10.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018886 2024-04-25T08:42:10.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018886' 2024-04-25T08:42:10.795 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 128849018886 2024-04-25T08:42:10.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:42:11.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018887 -lt 128849018886 2024-04-25T08:42:11.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:42:11.233 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:11.234 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:11.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:42:11.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:11.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:11.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:11.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:11.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:11.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:11.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:12.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:42:12.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:12.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:12.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:12.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:42:12.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:42:12.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:42:12.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2024-04-25T08:42:12.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:12.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2024-04-25T08:42:12.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:12.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2024-04-25T08:42:12.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:12.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2024-04-25T08:42:13.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:13.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2024-04-25T08:42:13.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:13.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2024-04-25T08:42:13.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:13.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2024-04-25T08:42:13.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:13.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2024-04-25T08:42:13.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:13.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2024-04-25T08:42:13.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:13.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2024-04-25T08:42:13.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:13.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2024-04-25T08:42:14.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:14.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2024-04-25T08:42:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:14.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2024-04-25T08:42:14.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:14.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2024-04-25T08:42:14.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:14.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2024-04-25T08:42:14.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:14.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2024-04-25T08:42:14.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:14.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2024-04-25T08:42:14.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:14.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2024-04-25T08:42:14.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:14.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2024-04-25T08:42:15.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:15.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2024-04-25T08:42:15.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:15.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2024-04-25T08:42:15.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:15.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2024-04-25T08:42:15.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:15.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2024-04-25T08:42:15.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:15.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2024-04-25T08:42:15.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:15.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2024-04-25T08:42:15.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:15.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2024-04-25T08:42:16.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:16.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2024-04-25T08:42:16.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:16.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2024-04-25T08:42:16.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:16.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2024-04-25T08:42:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:16.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2024-04-25T08:42:16.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:16.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2024-04-25T08:42:16.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:16.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2024-04-25T08:42:16.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:16.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2024-04-25T08:42:16.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:16.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2024-04-25T08:42:17.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:17.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2024-04-25T08:42:17.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:17.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2024-04-25T08:42:17.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:17.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2024-04-25T08:42:17.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:17.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2024-04-25T08:42:17.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:17.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2024-04-25T08:42:17.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:17.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2024-04-25T08:42:17.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:17.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2024-04-25T08:42:17.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:17.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2024-04-25T08:42:18.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:18.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2024-04-25T08:42:18.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:18.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2024-04-25T08:42:18.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:18.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2024-04-25T08:42:18.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:18.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2024-04-25T08:42:18.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:18.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2024-04-25T08:42:18.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:18.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2024-04-25T08:42:18.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:18.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2024-04-25T08:42:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:18.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2024-04-25T08:42:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2024-04-25T08:42:19.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:19.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2024-04-25T08:42:19.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:19.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2024-04-25T08:42:19.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:19.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2024-04-25T08:42:19.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:19.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2024-04-25T08:42:19.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:19.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2024-04-25T08:42:19.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:19.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2024-04-25T08:42:19.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:19.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2024-04-25T08:42:20.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:20.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2024-04-25T08:42:20.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:20.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2024-04-25T08:42:20.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:20.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2024-04-25T08:42:20.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:20.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2024-04-25T08:42:20.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:20.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2024-04-25T08:42:20.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:20.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2024-04-25T08:42:20.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:20.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2024-04-25T08:42:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:20.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2024-04-25T08:42:21.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:21.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2024-04-25T08:42:21.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:21.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2024-04-25T08:42:21.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:21.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2024-04-25T08:42:21.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:21.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2024-04-25T08:42:21.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:21.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2024-04-25T08:42:21.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:21.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2024-04-25T08:42:21.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:21.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2024-04-25T08:42:21.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:21.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2024-04-25T08:42:22.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:22.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2024-04-25T08:42:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:22.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2024-04-25T08:42:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:22.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2024-04-25T08:42:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:22.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2024-04-25T08:42:22.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:22.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2024-04-25T08:42:22.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:22.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2024-04-25T08:42:22.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:22.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2024-04-25T08:42:22.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:22.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2024-04-25T08:42:23.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:23.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2024-04-25T08:42:23.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:23.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2024-04-25T08:42:23.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:23.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2024-04-25T08:42:23.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:23.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2024-04-25T08:42:23.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:23.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2024-04-25T08:42:23.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:23.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2024-04-25T08:42:23.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:23.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2024-04-25T08:42:23.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:23.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2024-04-25T08:42:24.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:24.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2024-04-25T08:42:24.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:24.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2024-04-25T08:42:24.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:24.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2024-04-25T08:42:24.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:24.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2024-04-25T08:42:24.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:24.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2024-04-25T08:42:24.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:24.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2024-04-25T08:42:24.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:24.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2024-04-25T08:42:24.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:24.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2024-04-25T08:42:25.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:25.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2024-04-25T08:42:25.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:25.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2024-04-25T08:42:25.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:25.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2024-04-25T08:42:25.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:25.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2024-04-25T08:42:25.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:25.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2024-04-25T08:42:25.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:25.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2024-04-25T08:42:25.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:25.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2024-04-25T08:42:25.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:25.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2024-04-25T08:42:26.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:26.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2024-04-25T08:42:26.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:26.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2024-04-25T08:42:26.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:26.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2024-04-25T08:42:26.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:26.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2024-04-25T08:42:26.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:26.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2024-04-25T08:42:26.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:26.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2024-04-25T08:42:26.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:26.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2024-04-25T08:42:26.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:26.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2024-04-25T08:42:27.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:27.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2024-04-25T08:42:27.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:27.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2024-04-25T08:42:27.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:27.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2024-04-25T08:42:27.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:27.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2024-04-25T08:42:27.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:27.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2024-04-25T08:42:27.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:27.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2024-04-25T08:42:27.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:27.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2024-04-25T08:42:27.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:27.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2024-04-25T08:42:28.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2024-04-25T08:42:28.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2024-04-25T08:42:28.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2024-04-25T08:42:28.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2024-04-25T08:42:28.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2024-04-25T08:42:28.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2024-04-25T08:42:28.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2024-04-25T08:42:28.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2024-04-25T08:42:28.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:28.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2024-04-25T08:42:29.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:29.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2024-04-25T08:42:29.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:29.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2024-04-25T08:42:29.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:29.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2024-04-25T08:42:29.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:29.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2024-04-25T08:42:29.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:29.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2024-04-25T08:42:29.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:29.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2024-04-25T08:42:29.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:29.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2024-04-25T08:42:29.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:29.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2024-04-25T08:42:30.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:30.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2024-04-25T08:42:30.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:30.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2024-04-25T08:42:30.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:30.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2024-04-25T08:42:30.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:30.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2024-04-25T08:42:30.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:30.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2024-04-25T08:42:30.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:30.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2024-04-25T08:42:30.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:30.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2024-04-25T08:42:31.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2024-04-25T08:42:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2024-04-25T08:42:31.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2024-04-25T08:42:31.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2024-04-25T08:42:31.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2024-04-25T08:42:31.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2024-04-25T08:42:31.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2024-04-25T08:42:31.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2024-04-25T08:42:31.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:31.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2024-04-25T08:42:32.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:32.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2024-04-25T08:42:32.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:32.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2024-04-25T08:42:32.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:32.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2024-04-25T08:42:32.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:32.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2024-04-25T08:42:32.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:32.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2024-04-25T08:42:32.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:32.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2024-04-25T08:42:32.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:32.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2024-04-25T08:42:32.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:32.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2024-04-25T08:42:33.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:33.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2024-04-25T08:42:33.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:33.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2024-04-25T08:42:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2024-04-25T08:42:33.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:33.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2024-04-25T08:42:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2024-04-25T08:42:33.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:33.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2024-04-25T08:42:33.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:33.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2024-04-25T08:42:34.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:34.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2024-04-25T08:42:34.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:34.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2024-04-25T08:42:34.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:34.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2024-04-25T08:42:34.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:34.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2024-04-25T08:42:34.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:34.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2024-04-25T08:42:34.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:34.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2024-04-25T08:42:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:34.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2024-04-25T08:42:34.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:34.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2024-04-25T08:42:35.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:35.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2024-04-25T08:42:35.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:35.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2024-04-25T08:42:35.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:35.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2024-04-25T08:42:35.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:35.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2024-04-25T08:42:35.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:35.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2024-04-25T08:42:35.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:35.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2024-04-25T08:42:35.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:35.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2024-04-25T08:42:35.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:35.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2024-04-25T08:42:36.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2024-04-25T08:42:36.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2024-04-25T08:42:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2024-04-25T08:42:36.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2024-04-25T08:42:36.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2024-04-25T08:42:36.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2024-04-25T08:42:36.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2024-04-25T08:42:36.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2024-04-25T08:42:36.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:36.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2024-04-25T08:42:37.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:37.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2024-04-25T08:42:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:37.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2024-04-25T08:42:37.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:37.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2024-04-25T08:42:37.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:37.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2024-04-25T08:42:37.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:37.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2024-04-25T08:42:37.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:42:37.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2024-04-25T08:42:37.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2024-04-25T08:42:37.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:42:37.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:42:37.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:42:37.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:42:38.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2024-04-25T08:42:38.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2024-04-25T08:42:38.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:42:38.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:42:38.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:42:38.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:42:38.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2024-04-25T08:42:38.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2024-04-25T08:42:38.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:42:38.689 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:42:38.689 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:42:38.689 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:42:38.689 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:42:38.690 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:42:38.690 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:42:39.112 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:42:39.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:42:39.113 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:42:39.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2024-04-25T08:42:39.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2024-04-25T08:42:40.635 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is set 2024-04-25T08:42:40.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2024-04-25T08:42:41.179 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already out. osd.0 is already out. 2024-04-25T08:42:41.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2024-04-25T08:42:42.412 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 4 2024-04-25T08:42:42.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2024-04-25T08:42:44.128 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is unset 2024-04-25T08:42:44.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2024-04-25T08:42:44.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:42:44.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:42:44.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:42:44.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:42:44.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2024-04-25T08:42:44.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T08:42:44.744 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:42:44.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2024-04-25T08:42:46.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2024-04-25T08:42:46.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:42:46.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:42:46.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:42:46.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:42:46.762 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:42:46.762 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:42:46.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:42:46.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:42:46.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:42:46.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:42:46.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:42:46.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:42:46.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:42:46.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:42:46.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:47.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:42:47.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836537 2024-04-25T08:42:47.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836537 2024-04-25T08:42:47.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537' 2024-04-25T08:42:47.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:47.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:42:47.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673011 2024-04-25T08:42:47.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673011 2024-04-25T08:42:47.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011' 2024-04-25T08:42:47.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:47.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:42:47.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509484 2024-04-25T08:42:47.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509484 2024-04-25T08:42:47.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011 2-64424509484' 2024-04-25T08:42:47.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:47.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:42:48.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345950 2024-04-25T08:42:48.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345950 2024-04-25T08:42:48.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011 2-64424509484 3-85899345950' 2024-04-25T08:42:48.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:48.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:42:48.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182424 2024-04-25T08:42:48.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182424 2024-04-25T08:42:48.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011 2-64424509484 3-85899345950 4-107374182424' 2024-04-25T08:42:48.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:48.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:42:48.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018897 2024-04-25T08:42:48.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018897 2024-04-25T08:42:48.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673011 2-64424509484 3-85899345950 4-107374182424 5-128849018897' 2024-04-25T08:42:48.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:48.444 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836537 2024-04-25T08:42:48.444 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:48.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:42:48.445 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836537 2024-04-25T08:42:48.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:48.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836537 2024-04-25T08:42:48.447 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836537 2024-04-25T08:42:48.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-04-25T08:42:48.447 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:42:48.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836535 -lt 21474836537 2024-04-25T08:42:48.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:42:49.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:42:49.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:42:50.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836537 -lt 21474836537 2024-04-25T08:42:50.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:50.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673011 2024-04-25T08:42:50.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:50.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:42:50.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673011 2024-04-25T08:42:50.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:50.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673011 2024-04-25T08:42:50.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-04-25T08:42:50.275 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673011 2024-04-25T08:42:50.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:42:50.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673011 -lt 42949673011 2024-04-25T08:42:50.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:50.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509484 2024-04-25T08:42:50.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:50.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:42:50.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509484 2024-04-25T08:42:50.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:50.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509484 2024-04-25T08:42:50.805 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509484 2024-04-25T08:42:50.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-04-25T08:42:50.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:42:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509485 -lt 64424509484 2024-04-25T08:42:51.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:51.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345950 2024-04-25T08:42:51.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:51.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:42:51.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345950 2024-04-25T08:42:51.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:51.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345950 2024-04-25T08:42:51.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345950' 2024-04-25T08:42:51.323 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345950 2024-04-25T08:42:51.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:42:51.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345951 -lt 85899345950 2024-04-25T08:42:51.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:51.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182424 2024-04-25T08:42:51.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:51.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:42:51.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182424 2024-04-25T08:42:51.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:51.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182424 2024-04-25T08:42:51.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182424' 2024-04-25T08:42:51.768 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182424 2024-04-25T08:42:51.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:42:52.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182425 -lt 107374182424 2024-04-25T08:42:52.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:52.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018897 2024-04-25T08:42:52.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:52.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:42:52.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018897 2024-04-25T08:42:52.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:52.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018897 2024-04-25T08:42:52.285 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 128849018897 2024-04-25T08:42:52.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018897' 2024-04-25T08:42:52.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:42:52.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018897 -lt 128849018897 2024-04-25T08:42:52.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:42:52.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:52.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:42:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:53.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:53.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:53.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:53.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:53.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:53.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:53.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:53.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:53.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:54.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:54.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:42:54.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:42:54.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:42:54.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:54.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:54.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:54.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:54.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:54.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:54.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:54.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:54.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:54.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:55.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:55.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:55.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:55.003 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:55.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:55.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:55.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:55.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:55.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:55.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:55.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:55.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:55.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:42:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:42:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:56.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:56.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:56.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:56.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:56.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:56.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:56.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:56.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:56.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:56.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:56.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:56.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:56.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:56.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:57.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:57.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:57.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:57.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:57.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:57.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:57.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:57.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:57.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:58.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:42:58.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:42:58.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:58.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:58.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:58.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:58.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:58.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:58.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:58.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:58.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:58.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:58.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:58.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:58.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:58.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:58.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:59.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:59.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:59.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:59.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:59.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:59.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:59.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:59.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:59.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:59.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:42:59.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:42:59.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:59.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:59.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:00.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:00.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:00.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:00.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:00.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:00.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:00.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:00.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:00.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:00.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:00.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:00.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:01.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:01.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:01.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:01.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:01.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:01.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:01.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:01.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:01.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:01.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:01.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:01.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:01.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:01.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:01.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:01.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:01.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:01.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:01.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:01.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:01.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:01.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:02.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:02.116 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:02.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:02.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:02.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:02.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:02.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:02.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:02.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:02.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:02.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:02.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:02.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:03.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:03.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:03.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:03.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:03.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:03.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:03.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:03.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:03.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:03.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:03.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:03.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:03.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:04.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:04.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:04.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:04.029 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:04.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:04.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:04.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:04.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:04.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:04.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:04.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:04.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:04.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:05.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:05.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:05.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:05.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:05.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:05.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:05.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:05.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:05.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:05.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:05.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:05.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:05.844 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:06.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:06.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:06.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:06.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:06.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:06.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:06.530 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:07.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:07.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:07.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:07.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:07.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:07.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:07.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:07.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:07.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:07.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:07.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:07.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:07.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:07.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:08.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:08.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:08.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:08.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:08.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:08.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:08.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:08.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:08.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:08.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:08.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:08.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:08.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:08.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:08.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:08.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:08.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:08.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:08.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:08.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:08.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:08.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:09.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:09.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:09.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:09.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:09.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:09.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:09.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:09.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:09.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:09.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:09.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:09.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:09.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:10.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:10.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:10.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:10.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:10.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:10.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:10.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:10.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:10.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:10.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:10.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:10.621 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:10.621 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:11.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:11.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:11.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:11.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:11.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:11.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:11.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:11.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:11.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:11.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:11.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:11.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:11.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:12.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:12.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:12.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:12.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:12.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:12.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:12.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:12.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:12.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:12.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:12.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:12.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:12.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:12.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:12.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:12.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:12.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:13.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:13.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:14.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:14.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:14.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:14.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:14.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:14.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:14.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:14.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:14.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:14.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:14.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:14.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:14.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:14.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:14.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:14.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:14.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:15.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:15.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:15.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:15.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:15.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:15.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:15.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:15.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:15.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:15.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:15.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:15.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:15.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:15.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:16.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:16.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:16.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:16.216 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:16.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:16.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:16.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:16.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:16.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:17.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:17.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:17.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:17.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:17.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:17.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:17.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:17.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:17.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:17.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:17.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:17.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:17.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:18.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:18.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:18.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:18.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:18.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:18.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:18.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:18.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:18.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:18.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:18.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:18.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:18.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:19.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:19.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:19.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:19.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:19.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:19.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:19.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:19.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:19.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:19.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:19.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:19.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:19.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:19.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:19.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:19.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:20.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:21.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:21.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:21.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:21.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:21.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:21.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:21.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:21.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:21.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:21.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:21.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:21.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:21.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:22.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:22.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:22.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:22.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:22.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:22.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:22.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:22.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:22.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:22.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:22.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:22.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:22.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:22.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:22.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:22.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:22.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:23.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:23.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:23.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:23.454 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:24.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:24.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:24.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:24.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:24.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:24.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:24.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:24.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:24.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:24.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:24.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:24.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:25.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:25.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:25.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:25.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:25.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:25.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:25.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:25.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:25.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:25.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:25.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:25.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:26.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:26.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:26.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:26.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:26.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:26.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:26.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:26.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:26.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:26.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:26.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:26.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:26.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:26.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:26.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:26.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:26.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:27.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:27.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:27.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:27.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:27.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:27.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:27.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:27.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:27.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:27.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:27.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:27.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:27.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:28.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:43:28.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:43:28.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:28.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:28.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:28.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:28.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:28.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:28.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:28.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:28.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:28.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:28.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:28.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:28.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:28.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:28.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:29.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:29.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:29.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:29.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:29.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:29.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:29.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:29.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:29.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:30.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:30.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:30.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:30.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:30.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:30.472 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:30.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:30.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:30.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:30.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:30.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:30.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:30.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:30.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:30.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:31.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:31.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:31.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:31.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:31.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:31.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:31.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:31.393 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:31.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:31.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:31.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:31.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:31.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:31.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:32.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:32.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:32.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:32.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:32.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:32.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:32.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:32.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:32.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:32.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:32.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:32.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:33.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:33.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:33.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:33.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:33.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:33.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:33.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:33.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:33.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:43:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:43:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:33.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:33.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:33.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:33.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:33.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:33.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:33.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:33.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:33.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:33.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:34.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:43:34.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:34.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:34.156 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:34.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:43:34.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:43:34.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:43:34.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:43:34.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:43:34.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:43:34.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:43:34.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:43:34.682 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:43:35.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-04-25T08:43:35.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-04-25T08:43:35.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:43:35.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:43:35.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:43:35.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:43:35.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:43:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:43:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:43:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:43:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:43:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:43:35.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:43:35.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:43:35.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:43:35.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:43:35.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:43:36.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:43:36.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:43:36.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:43:36.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2024-04-25T08:43:36.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2024-04-25T08:43:36.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2024-04-25T08:43:36.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2024-04-25T08:43:36.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2024-04-25T08:43:36.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:43:36.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:43:36.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2024-04-25T08:43:36.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:43:36.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2024-04-25T08:43:36.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:43:36.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-04-25T08:43:36.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:43:36.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2024-04-25T08:43:36.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-04-25T08:43:36.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:43:36.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:43:36.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:43:36.314 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.314 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:43:36.314 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:43:36.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2024-04-25T08:43:36.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:43:36.343 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:36.343 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.343 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:43:36.343 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:43:36.343 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:43:36.343 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:36.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2024-04-25T08:43:36.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2024-04-25T08:43:36.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T08:43:36.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:43:36.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:43:36.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:43:36.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:36.348 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.348 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:43:36.348 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:36.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:43:36.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:43:36.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:43:36.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:43:36.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:43:36.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:43:36.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:43:36.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:36.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:43:36.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:43:36.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:43:36.359 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:36.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-04-25T08:43:36.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-04-25T08:43:36.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T08:43:36.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:43:36.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:43:36.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:43:36.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:36.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:43:36.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:36.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:43:36.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:43:36.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:43:36.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:43:36.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:43:36.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:43:36.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2024-04-25T08:43:36.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2024-04-25T08:43:36.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:36.374 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.374 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:43:36.374 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-04-25T08:43:36.374 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-04-25T08:43:36.374 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:36.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2024-04-25T08:43:36.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2024-04-25T08:43:36.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:43:36.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:43:36.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:43:36.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:43:36.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:43:36.383 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:43:36.383 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-04-25T08:43:36.383 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:43:36.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-04-25T08:43:36.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-04-25T08:43:36.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:43:36.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:43:36.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:43:36.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:43:36.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2024-04-25T08:43:36.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:43:36.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:43:36.914 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T08:43:36.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2024-04-25T08:43:36.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:43:36.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:43:36.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:43:36.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:43:36.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:43:42.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:43:42.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:43:42.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:43:42.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:43:42.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:43:42.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:43:42.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:43:42.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:43:42.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:43:42.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:43:42.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:43:42.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:43:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:43:42.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:43:42.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:43:42.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:43:42.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:43:42.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:42.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:43:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:43:42.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:43:42.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:43:42.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:43:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:43:42.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:43:42.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:43:42.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:43:42.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:43:42.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:43:42.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:43:42.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:43:42.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:43:42.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:43:42.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:43:42.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:43:42.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:43:42.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:43:42.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:43:42.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:43:42.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:42.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:43:42.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:43:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:43:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:43:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:43:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:43:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:43:42.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.274961 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:43:42.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:43:42.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:43:42.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:43:42.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:43:42.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:42.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:42.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:42.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:42.811 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:43:42.811 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:43:42.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:43:43.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:43:43.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:43:43.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:43:43.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:43:43.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:43:43.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:43:43.007 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:43:43.007 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:43:43.007 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:43:43.008 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:43:43.008 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:43.008 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:43:43.008 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:43:43.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:43:43.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get fsid 2024-04-25T08:43:43.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:43:43.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:43:43.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:43:43.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:43:43.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:43:43.128 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:43:43.128 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:43:43.128 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:43:43.128 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:43:43.128 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:43.129 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:43:43.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.274961/ceph-mon.a.asok 2024-04-25T08:43:43.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:43:43.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.274961/ceph-mon.a.asok config get mon_host 2024-04-25T08:43:43.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2024-04-25T08:43:43.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:43:43.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:43:43.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:43:43.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:43:43.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:43:43.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:43:43.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:43:43.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:43.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:43.788 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:43.788 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:43.788 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:43:43.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:43:43.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:43:43.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:43:43.824 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2024-04-25T08:43:43.825 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2024-04-25T08:43:43.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:43:43.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2024-04-25T08:43:43.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:43:43.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:43:43.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:43:43.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:43:43.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a63d8f22-fe5a-42c6-a976-67aeff0fa6f0 2024-04-25T08:43:43.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a63d8f22-fe5a-42c6-a976-67aeff0fa6f0' 2024-04-25T08:43:43.833 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 a63d8f22-fe5a-42c6-a976-67aeff0fa6f0 2024-04-25T08:43:43.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:43:43.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/FypmOe4cMxAAMMPpoAAXJHV3tUxLH1CAHQ== 2024-04-25T08:43:43.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/FypmOe4cMxAAMMPpoAAXJHV3tUxLH1CAHQ=="}' 2024-04-25T08:43:43.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a63d8f22-fe5a-42c6-a976-67aeff0fa6f0 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:43:44.143 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:43:44.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:43:44.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/FypmOe4cMxAAMMPpoAAXJHV3tUxLH1CAHQ== --osd-uuid a63d8f22-fe5a-42c6-a976-67aeff0fa6f0 2024-04-25T08:43:44.193 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:44.199+0000 7fa7740007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:44.211 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:44.215+0000 7fa7740007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:44.212 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:44.219+0000 7fa7740007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:44.219+0000 7fa7740007c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:44.219+0000 7fa7740007c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:44.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:44.219+0000 7fa7740007c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:44.251 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:44.255+0000 7fa7740007c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:43:50.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:43:50.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:43:50.307 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:43:50.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:43:50.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:43:50.869 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:43:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:43:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:43:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:43:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:43:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:43:50.903 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:50.907+0000 7f316fff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:50.905 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:50.911+0000 7f316fff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:50.906 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:50.911+0000 7f316fff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:51.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:43:51.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:43:51.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:43:51.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:43:51.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:43:51.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:51.262 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:43:51.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:43:51.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:51.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:51.470 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:51.475+0000 7f316fff27c0 -1 Falling back to public interface 2024-04-25T08:43:51.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:52.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:52.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:52.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:43:52.675 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:43:52.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:52.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:53.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:53.973 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:43:53.979+0000 7f316fff27c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:43:54.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:54.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:54.090 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:43:54.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:43:54.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:54.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:54.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:55.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:55.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:55.570 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:43:55.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:43:55.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:55.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:56.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:57.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:57.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:57.026 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:43:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:43:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:57.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:58.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:58.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:58.456 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:43:58.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:43:58.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:58.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:58.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:59.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:59.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:59.999 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:43:59.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:43:59.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:59.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:00.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:01.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:01.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:01.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:44:01.433 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:44:01.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:01.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:02.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:03.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:03.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:03.014 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:44:03.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:44:03.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:03.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:03.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:04.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:04.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:04.465 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:44:04.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:44:04.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:04.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:05.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:06.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:06.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:06.227 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:44:06.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:44:06.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:06.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:06.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:07.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:07.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:07.666 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:44:07.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:44:07.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:07.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:08.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:09.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:09.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:09.630 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:44:09.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:44:09.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:09.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:10.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:11.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:11.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:11.583 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:44:11.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:44:11.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:11.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:12.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:13.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:44:13.553 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:44:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:14.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:15.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:15.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:15.503 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:44:15.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:44:15.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:15.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:44:15.909 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2250584808,v1:127.0.0.1:6803/2250584808] [v2:127.0.0.1:6804/2250584808,v1:127.0.0.1:6805/2250584808] exists,up a63d8f22-fe5a-42c6-a976-67aeff0fa6f0 2024-04-25T08:44:15.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:44:15.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:44:15.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:44:15.915 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:44:15.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:44:15.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:44:15.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:44:15.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c816f05b-3051-42b3-ac61-6a5d6188a9d2 2024-04-25T08:44:15.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c816f05b-3051-42b3-ac61-6a5d6188a9d2' 2024-04-25T08:44:15.918 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 c816f05b-3051-42b3-ac61-6a5d6188a9d2 2024-04-25T08:44:15.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:44:15.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfFypmmIeBOBAAYTikDEUB1UTepUHWDP82dw== 2024-04-25T08:44:15.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfFypmmIeBOBAAYTikDEUB1UTepUHWDP82dw=="}' 2024-04-25T08:44:15.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c816f05b-3051-42b3-ac61-6a5d6188a9d2 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:44:16.473 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:44:16.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:44:16.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfFypmmIeBOBAAYTikDEUB1UTepUHWDP82dw== --osd-uuid c816f05b-3051-42b3-ac61-6a5d6188a9d2 2024-04-25T08:44:16.528 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:16.534+0000 7efc3b8497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:16.547 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:16.550+0000 7efc3b8497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:16.548 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:16.554+0000 7efc3b8497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:16.548 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:16.554+0000 7efc3b8497c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:44:16.549 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:16.554+0000 7efc3b8497c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:44:16.549 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:16.554+0000 7efc3b8497c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:44:16.644 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:16.650+0000 7efc3b8497c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:44:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:44:22.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:44:22.434 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:44:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:44:22.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:44:23.000 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:44:23.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:44:23.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:44:23.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:44:23.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:44:23.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:44:23.030 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:23.034+0000 7fe0a00b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:23.032 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:23.038+0000 7fe0a00b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:23.033 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:23.038+0000 7fe0a00b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:23.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:44:23.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:44:23.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:44:23.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:44:23.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:44:23.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:23.427 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:44:23.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:44:23.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:23.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:23.609 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:23.614+0000 7fe0a00b67c0 -1 Falling back to public interface 2024-04-25T08:44:23.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:24.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:24.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:24.851 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:44:24.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:44:24.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:24.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:25.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:26.126 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:26.130+0000 7fe0a00b67c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:44:26.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:26.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:26.263 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:44:26.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:44:26.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:26.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:26.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:27.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:27.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:27.820 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:44:27.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:44:27.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:27.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:28.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:29.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:29.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:29.255 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:44:29.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:44:29.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:29.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:29.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:30.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:30.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:30.678 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:44:30.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:44:30.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:30.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:31.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:32.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:32.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:32.291 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:44:32.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:44:32.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:32.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:32.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:33.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:33.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:33.723 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:44:33.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:44:33.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:33.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:34.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:35.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:35.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:35.284 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:44:35.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:44:35.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:35.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:35.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:36.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:36.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:36.755 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:44:36.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:44:36.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:36.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:37.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:38.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:38.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:38.319 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:44:38.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:44:38.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:38.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:38.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:39.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:39.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:39.785 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:44:39.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:44:39.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:39.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:40.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:41.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:41.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:41.721 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:44:41.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:44:41.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:41.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:42.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:43.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:43.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:43.510 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:44:43.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:44:43.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:43.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:44.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:45.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:45.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:45.251 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:44:45.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:44:45.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:45.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:46.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:47.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:47.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:47.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:44:47.243 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:44:47.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:47.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:47.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:48.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:48.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:48.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:44:48.984 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:44:48.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:48.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:49.467 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3647083801,v1:127.0.0.1:6811/3647083801] [v2:127.0.0.1:6812/3647083801,v1:127.0.0.1:6813/3647083801] exists,up c816f05b-3051-42b3-ac61-6a5d6188a9d2 2024-04-25T08:44:49.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:44:49.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:44:49.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:44:49.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:44:49.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:44:49.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:44:49.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:44:49.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:44:49.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:44:49.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:44:49.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=705b5df3-eb56-4d0d-bba2-22898e340a32 2024-04-25T08:44:49.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 705b5df3-eb56-4d0d-bba2-22898e340a32' 2024-04-25T08:44:49.474 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 705b5df3-eb56-4d0d-bba2-22898e340a32 2024-04-25T08:44:49.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:44:49.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABGCpmL9f9HRAAt+PNflVZ2oshNWPn+QeidQ== 2024-04-25T08:44:49.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABGCpmL9f9HRAAt+PNflVZ2oshNWPn+QeidQ=="}' 2024-04-25T08:44:49.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 705b5df3-eb56-4d0d-bba2-22898e340a32 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:44:49.966 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:44:49.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:44:49.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQABGCpmL9f9HRAAt+PNflVZ2oshNWPn+QeidQ== --osd-uuid 705b5df3-eb56-4d0d-bba2-22898e340a32 2024-04-25T08:44:50.017 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:50.026+0000 7faacf7237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:50.018 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:50.026+0000 7faacf7237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:50.020 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:50.030+0000 7faacf7237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:50.020 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:50.030+0000 7faacf7237c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:50.020 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:50.030+0000 7faacf7237c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:50.020 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:50.030+0000 7faacf7237c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:50.065 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:50.074+0000 7faacf7237c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:44:55.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:44:55.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:44:55.910 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:44:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:44:55.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:44:56.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:44:56.473 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:44:56.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d02c8e66-5498-447d-b427-aea25d463b77 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.274961/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:44:56.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:44:56.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:44:56.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:44:56.503 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:56.510+0000 7f57250777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:56.520 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:56.530+0000 7f57250777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:56.522 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:56.530+0000 7f57250777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:56.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:44:56.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:44:56.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:44:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:44:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:44:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:44:56.893 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:44:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:57.089 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:57.098+0000 7f57250777c0 -1 Falling back to public interface 2024-04-25T08:44:57.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:58.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:58.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:58.341 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:44:58.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:44:58.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:58.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:58.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:59.645 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:44:59.653+0000 7f57250777c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:44:59.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:59.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:59.820 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:44:59.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:44:59.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:59.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:00.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:01.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:01.366 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:45:01.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:45:01.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:01.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:01.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:02.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:02.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:02.795 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:45:02.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:45:02.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:02.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:03.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:04.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:04.274 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:45:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:45:04.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:04.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:04.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:05.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:05.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:05.783 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:45:05.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:45:05.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:05.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:06.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:07.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:07.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:07.192 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:45:07.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:45:07.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:07.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:07.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:08.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:08.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:08.774 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:45:08.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:45:08.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:08.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:09.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:10.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:10.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:10.339 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:45:10.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:45:10.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:10.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:10.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:11.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:11.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:11.839 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:45:11.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:45:11.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:11.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:13.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:13.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:13.256 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:45:13.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:45:13.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:13.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:14.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:15.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:15.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:15.234 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:45:15.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:45:15.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:15.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:16.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:17.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:17.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:17.257 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:45:17.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:45:17.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:17.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:18.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:19.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:19.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:19.370 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:45:19.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:45:19.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:19.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:20.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:21.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:21.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:21.392 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:45:21.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:45:21.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:21.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:23.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:23.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:23.298 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:45:23.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:45:23.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:23.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:45:23.754 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3215149449,v1:127.0.0.1:6819/3215149449] [v2:127.0.0.1:6820/3215149449,v1:127.0.0.1:6821/3215149449] exists,up 705b5df3-eb56-4d0d-bba2-22898e340a32 2024-04-25T08:45:23.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:45:23.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:45:23.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:45:23.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2024-04-25T08:45:23.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:45:24.364 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T08:45:24.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:45:25.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2024-04-25T08:45:26.201 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 1 2024-04-25T08:45:26.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2024-04-25T08:45:26.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:45:26.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:45:26.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:45:26.225 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:45:26.226 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:45:26.226 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:45:26.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:45:26.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:45:26.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:45:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:45:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:45:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:45:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:45:26.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:45:26.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:45:26.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:45:26.875 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:45:26.875 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T08:45:26.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:45:26.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:45:26.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:45:27.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836499 2024-04-25T08:45:27.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836499 2024-04-25T08:45:27.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499' 2024-04-25T08:45:27.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:45:27.031 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:45:27.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672973 2024-04-25T08:45:27.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672973 2024-04-25T08:45:27.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672973' 2024-04-25T08:45:27.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:45:27.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:45:27.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-04-25T08:45:27.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-04-25T08:45:27.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836499 1-42949672973 2-64424509451' 2024-04-25T08:45:27.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:45:27.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836499 2024-04-25T08:45:27.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:45:27.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:45:27.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836499 2024-04-25T08:45:27.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:45:27.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836499 2024-04-25T08:45:27.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-04-25T08:45:27.337 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836499 2024-04-25T08:45:27.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:45:27.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836497 -lt 21474836499 2024-04-25T08:45:27.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:45:28.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:45:28.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:45:29.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836499 2024-04-25T08:45:29.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:45:29.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672973 2024-04-25T08:45:29.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:45:29.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:45:29.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672973 2024-04-25T08:45:29.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:45:29.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672973 2024-04-25T08:45:29.171 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672973 2024-04-25T08:45:29.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-04-25T08:45:29.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:45:29.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672973 2024-04-25T08:45:29.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:45:29.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:45:29.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-04-25T08:45:29.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:45:29.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-04-25T08:45:29.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:45:29.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-04-25T08:45:29.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-25T08:45:29.658 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509451 2024-04-25T08:45:29.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:45:30.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509451 2024-04-25T08:45:30.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:45:30.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:30.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:30.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:45:30.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:30.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:30.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:30.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:30.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:30.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:30.690 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:31.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:45:31.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:31.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:31.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:31.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:45:31.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:45:31.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:45:31.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2024-04-25T08:45:31.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:31.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2024-04-25T08:45:31.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:31.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2024-04-25T08:45:31.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:31.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2024-04-25T08:45:31.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:31.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2024-04-25T08:45:32.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2024-04-25T08:45:32.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2024-04-25T08:45:32.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2024-04-25T08:45:32.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2024-04-25T08:45:32.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2024-04-25T08:45:32.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2024-04-25T08:45:32.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2024-04-25T08:45:32.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2024-04-25T08:45:32.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2024-04-25T08:45:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2024-04-25T08:45:32.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2024-04-25T08:45:32.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2024-04-25T08:45:32.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2024-04-25T08:45:32.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2024-04-25T08:45:32.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:32.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2024-04-25T08:45:33.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2024-04-25T08:45:33.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2024-04-25T08:45:33.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2024-04-25T08:45:33.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2024-04-25T08:45:33.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2024-04-25T08:45:33.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2024-04-25T08:45:33.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2024-04-25T08:45:33.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2024-04-25T08:45:33.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2024-04-25T08:45:33.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2024-04-25T08:45:33.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2024-04-25T08:45:33.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2024-04-25T08:45:33.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:33.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2024-04-25T08:45:34.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2024-04-25T08:45:34.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2024-04-25T08:45:34.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2024-04-25T08:45:34.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2024-04-25T08:45:34.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2024-04-25T08:45:34.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2024-04-25T08:45:34.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2024-04-25T08:45:34.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2024-04-25T08:45:34.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2024-04-25T08:45:34.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2024-04-25T08:45:34.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2024-04-25T08:45:34.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2024-04-25T08:45:34.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:34.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2024-04-25T08:45:35.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2024-04-25T08:45:35.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2024-04-25T08:45:35.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2024-04-25T08:45:35.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2024-04-25T08:45:35.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2024-04-25T08:45:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2024-04-25T08:45:35.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2024-04-25T08:45:35.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2024-04-25T08:45:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2024-04-25T08:45:35.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2024-04-25T08:45:35.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2024-04-25T08:45:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2024-04-25T08:45:35.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2024-04-25T08:45:35.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2024-04-25T08:45:35.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:35.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2024-04-25T08:45:36.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2024-04-25T08:45:36.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2024-04-25T08:45:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2024-04-25T08:45:36.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2024-04-25T08:45:36.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2024-04-25T08:45:36.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2024-04-25T08:45:36.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2024-04-25T08:45:36.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2024-04-25T08:45:36.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2024-04-25T08:45:36.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2024-04-25T08:45:36.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2024-04-25T08:45:36.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2024-04-25T08:45:36.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:36.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2024-04-25T08:45:37.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2024-04-25T08:45:37.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2024-04-25T08:45:37.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2024-04-25T08:45:37.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2024-04-25T08:45:37.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2024-04-25T08:45:37.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2024-04-25T08:45:37.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2024-04-25T08:45:37.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2024-04-25T08:45:37.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2024-04-25T08:45:37.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2024-04-25T08:45:37.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2024-04-25T08:45:37.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2024-04-25T08:45:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2024-04-25T08:45:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2024-04-25T08:45:37.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:37.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2024-04-25T08:45:38.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2024-04-25T08:45:38.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2024-04-25T08:45:38.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2024-04-25T08:45:38.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2024-04-25T08:45:38.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2024-04-25T08:45:38.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2024-04-25T08:45:38.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2024-04-25T08:45:38.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2024-04-25T08:45:38.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2024-04-25T08:45:38.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2024-04-25T08:45:38.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2024-04-25T08:45:38.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2024-04-25T08:45:38.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2024-04-25T08:45:38.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:38.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2024-04-25T08:45:39.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2024-04-25T08:45:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2024-04-25T08:45:39.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2024-04-25T08:45:39.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2024-04-25T08:45:39.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2024-04-25T08:45:39.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2024-04-25T08:45:39.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2024-04-25T08:45:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2024-04-25T08:45:39.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2024-04-25T08:45:39.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2024-04-25T08:45:39.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2024-04-25T08:45:39.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2024-04-25T08:45:39.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2024-04-25T08:45:39.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:39.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2024-04-25T08:45:40.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2024-04-25T08:45:40.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2024-04-25T08:45:40.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2024-04-25T08:45:40.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2024-04-25T08:45:40.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2024-04-25T08:45:40.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2024-04-25T08:45:40.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2024-04-25T08:45:40.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2024-04-25T08:45:40.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2024-04-25T08:45:40.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2024-04-25T08:45:40.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2024-04-25T08:45:40.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2024-04-25T08:45:40.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2024-04-25T08:45:40.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:40.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2024-04-25T08:45:41.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2024-04-25T08:45:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2024-04-25T08:45:41.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2024-04-25T08:45:41.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2024-04-25T08:45:41.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2024-04-25T08:45:41.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2024-04-25T08:45:41.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2024-04-25T08:45:41.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2024-04-25T08:45:41.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2024-04-25T08:45:41.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2024-04-25T08:45:41.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2024-04-25T08:45:41.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2024-04-25T08:45:41.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2024-04-25T08:45:41.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:41.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2024-04-25T08:45:42.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2024-04-25T08:45:42.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2024-04-25T08:45:42.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2024-04-25T08:45:42.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2024-04-25T08:45:42.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2024-04-25T08:45:42.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2024-04-25T08:45:42.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2024-04-25T08:45:42.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2024-04-25T08:45:42.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2024-04-25T08:45:42.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2024-04-25T08:45:42.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2024-04-25T08:45:42.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2024-04-25T08:45:42.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2024-04-25T08:45:42.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:42.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2024-04-25T08:45:43.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2024-04-25T08:45:43.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2024-04-25T08:45:43.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2024-04-25T08:45:43.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2024-04-25T08:45:43.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2024-04-25T08:45:43.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2024-04-25T08:45:43.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2024-04-25T08:45:43.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2024-04-25T08:45:43.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2024-04-25T08:45:43.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2024-04-25T08:45:43.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2024-04-25T08:45:43.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2024-04-25T08:45:43.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2024-04-25T08:45:43.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2024-04-25T08:45:43.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:43.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2024-04-25T08:45:44.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2024-04-25T08:45:44.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2024-04-25T08:45:44.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2024-04-25T08:45:44.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2024-04-25T08:45:44.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2024-04-25T08:45:44.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2024-04-25T08:45:44.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2024-04-25T08:45:44.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2024-04-25T08:45:44.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj182 /dev/null 2024-04-25T08:45:44.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj183 /dev/null 2024-04-25T08:45:44.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj184 /dev/null 2024-04-25T08:45:44.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj185 /dev/null 2024-04-25T08:45:44.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:44.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj186 /dev/null 2024-04-25T08:45:45.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj187 /dev/null 2024-04-25T08:45:45.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj188 /dev/null 2024-04-25T08:45:45.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj189 /dev/null 2024-04-25T08:45:45.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj190 /dev/null 2024-04-25T08:45:45.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj191 /dev/null 2024-04-25T08:45:45.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj192 /dev/null 2024-04-25T08:45:45.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj193 /dev/null 2024-04-25T08:45:45.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj194 /dev/null 2024-04-25T08:45:45.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj195 /dev/null 2024-04-25T08:45:45.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj196 /dev/null 2024-04-25T08:45:45.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj197 /dev/null 2024-04-25T08:45:45.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj198 /dev/null 2024-04-25T08:45:45.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj199 /dev/null 2024-04-25T08:45:45.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T08:45:45.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj200 /dev/null 2024-04-25T08:45:45.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: get_primary test obj1 2024-04-25T08:45:45.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:45:45.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:45:45.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:45:45.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:45:46.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: local primary=1 2024-04-25T08:45:46.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: get_pg test obj1 2024-04-25T08:45:46.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:45:46.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:45:46.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:45:46.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:45:46.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: local PG=1.0 2024-04-25T08:45:46.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:329: TEST_recovery_undersized: ceph osd set norecover 2024-04-25T08:45:47.522 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is set 2024-04-25T08:45:47.541 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: expr 3 - 1 2024-04-25T08:45:47.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: seq 0 2 2024-04-25T08:45:47.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:45:47.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:333: TEST_recovery_undersized: '[' 0 = 1 ']' 2024-04-25T08:45:47.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:337: TEST_recovery_undersized: ceph osd out osd.0 2024-04-25T08:45:48.192 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T08:45:48.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:338: TEST_recovery_undersized: break 2024-04-25T08:45:48.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:340: TEST_recovery_undersized: ceph osd pool set test size 4 2024-04-25T08:45:49.028 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 4 2024-04-25T08:45:49.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:341: TEST_recovery_undersized: ceph osd unset norecover 2024-04-25T08:45:49.866 INFO:tasks.workunit.client.0.smithi088.stderr:norecover is unset 2024-04-25T08:45:49.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: get_primary test obj1 2024-04-25T08:45:49.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:45:49.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:45:49.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:45:49.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:45:50.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:45:50.617 INFO:tasks.workunit.client.0.smithi088.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:45:50.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:344: TEST_recovery_undersized: sleep 10 2024-04-25T08:46:00.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:345: TEST_recovery_undersized: flush_pg_stats 2024-04-25T08:46:00.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:46:00.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:46:01.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:46:01.055 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:46:01.055 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T08:46:01.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:46:01.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:01.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:46:01.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-04-25T08:46:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-04-25T08:46:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-04-25T08:46:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:01.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:46:01.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-04-25T08:46:01.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-04-25T08:46:01.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-04-25T08:46:01.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:01.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:46:01.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-04-25T08:46:01.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-04-25T08:46:01.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509460' 2024-04-25T08:46:01.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:01.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-04-25T08:46:01.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:01.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:46:01.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-04-25T08:46:01.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:01.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-04-25T08:46:01.498 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836508 2024-04-25T08:46:01.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-25T08:46:01.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:46:01.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-04-25T08:46:01.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:46:02.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:46:02.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:46:03.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-04-25T08:46:03.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:03.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-04-25T08:46:03.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:03.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:46:03.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-04-25T08:46:03.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:03.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-04-25T08:46:03.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-25T08:46:03.327 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672982 2024-04-25T08:46:03.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:46:03.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-25T08:46:03.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:03.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-04-25T08:46:03.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:03.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:46:03.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-04-25T08:46:03.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:03.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-04-25T08:46:03.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-04-25T08:46:03.796 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509460 2024-04-25T08:46:03.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:46:04.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509460 -lt 64424509460 2024-04-25T08:46:04.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: seq 1 300 2024-04-25T08:46:04.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 300) 2024-04-25T08:46:04.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2024-04-25T08:46:04.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2024-04-25T08:46:04.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2024-04-25T08:46:04.597 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T08:46:04.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:354: TEST_recovery_undersized: break 2024-04-25T08:46:04.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: get_primary test obj1 2024-04-25T08:46:04.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:46:04.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:46:04.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:46:04.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:46:05.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: primary=1 2024-04-25T08:46:05.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:366: TEST_recovery_undersized: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:46:05.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: expr 200 '*' 3 2024-04-25T08:46:05.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: local first_degraded=600 2024-04-25T08:46:05.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: expr 200 '*' 2 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: local last_degraded=400 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:370: TEST_recovery_undersized: check td/osd-recovery-stats 1.0 1 replicated 600 400 0 0 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=600 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=400 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:46:05.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:46:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:46:05.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2]' 2024-04-25T08:46:05.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:46:05.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:46:05.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:46:05.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2] ' 2024-04-25T08:46:05.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:46:05.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:46:05.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=600 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 600 600 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=600 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=600 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:46:05.076 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=400 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 400 400 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=400 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=400 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2] ' 2024-04-25T08:46:05.085 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:46:05.086 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:46:05.086 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:46:05.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:46:05.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:46:05.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:46:05.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:46:05.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:46:05.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:46:05.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:46:05.091 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:46:05.091 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:46:05.091 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:46:05.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:46:05.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:46:05.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:46:05.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:46:05.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:46:05.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:46:05.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:46:05.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:372: TEST_recovery_undersized: delete_pool test 2024-04-25T08:46:05.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:46:05.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:46:05.603 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T08:46:05.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:373: TEST_recovery_undersized: kill_daemons td/osd-recovery-stats 2024-04-25T08:46:05.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:46:05.620 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:46:05.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:46:05.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:46:05.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:46:10.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:46:10.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:46:10.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:46:10.947 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:46:10.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:46:10.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:46:10.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:46:10.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:46:10.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:46:10.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:46:10.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:46:10.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:46:10.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:46:10.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:46:10.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:46:10.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:46:10.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:46:10.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:10.977 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:46:10.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-stats 0 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:46:10.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:46:10.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:46:10.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:46:10.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:46:10.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:46:10.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:46:10.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:46:10.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:46:10.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:46:10.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:46:10.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:46:10.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:46:10.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:46:10.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:46:10.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:46:10.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:46:10.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:46:10.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:46:10.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:10.988 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.274961 2024-04-25T08:46:10.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.274961 2024-04-25T08:46:10.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:46:10.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:46:10.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:46:10.990 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:46:10.990 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:46:11.041 INFO:tasks.workunit:Running workunit osd/osd-rep-recov-eio.sh... 2024-04-25T08:46:11.041 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-rep-recov-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2024-04-25T08:46:11.093 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:46:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:46:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:46:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:46:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:46:11.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:25: run: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:26: run: shift 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: export CEPH_MON=127.0.0.1:7140 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: CEPH_MON=127.0.0.1:7140 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:29: run: export CEPH_ARGS 2024-04-25T08:46:11.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: uuidgen 2024-04-25T08:46:11.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: CEPH_ARGS+='--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none ' 2024-04-25T08:46:11.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:31: run: CEPH_ARGS+='--mon-host=127.0.0.1:7140 ' 2024-04-25T08:46:11.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: set 2024-04-25T08:46:11.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:46:11.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: local 'funcs=TEST_rados_get_with_eio 2024-04-25T08:46:11.105 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_rados_repair_warning 2024-04-25T08:46:11.105 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_rep_backfill_unfound 2024-04-25T08:46:11.105 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_rep_recovery_unfound' 2024-04-25T08:46:11.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T08:46:11.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:46:11.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:46:11.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:46:11.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:46:11.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:46:11.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:46:11.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:46:11.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:46:11.110 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:46:11.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:46:11.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:46:11.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:46:11.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:46:11.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:46:11.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:46:11.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:46:11.115 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:11.115 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:46:11.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T08:46:11.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:46:11.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:46:11.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:46:11.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:11.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:46:11.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.389694 2024-04-25T08:46:11.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T08:46:11.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T08:46:11.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:46:11.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:46:11.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:46:11.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:46:11.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:11.590 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:46:11.590 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:46:11.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T08:46:11.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:46:11.818 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:46:11.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:46:11.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:46:11.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:46:11.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:46:11.820 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:46:11.820 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:46:11.820 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:46:11.820 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:46:11.820 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:11.821 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:46:11.821 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-mon.a.asok 2024-04-25T08:46:11.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:46:11.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.389694/ceph-mon.a.asok config get fsid 2024-04-25T08:46:11.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:46:11.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:46:11.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:46:11.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:46:11.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:46:11.938 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:46:11.938 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:46:11.938 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:46:11.938 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:46:11.938 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:11.938 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:46:11.938 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-mon.a.asok 2024-04-25T08:46:11.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:46:11.939 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.389694/ceph-mon.a.asok config get mon_host 2024-04-25T08:46:12.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T08:46:12.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:46:12.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:46:12.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:46:12.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:46:12.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T08:46:12.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:46:12.928 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:46:12.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:46:12.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:46:12.929 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:46:12.929 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:12.929 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:46:12.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:46:12.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:46:12.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:46:12.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T08:46:13.237 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T08:46:13.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_get_with_eio td/osd-rep-recov-eio.sh 2024-04-25T08:46:13.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:168: TEST_rados_get_with_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:46:13.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:170: TEST_rados_get_with_eio: setup_osds 4 2024-04-25T08:46:13.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-04-25T08:46:13.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T08:46:13.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T08:46:13.254 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-04-25T08:46:13.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-04-25T08:46:13.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:46:13.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:46:13.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:46:13.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:46:13.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:46:13.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:46:13.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:46:13.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:46:13.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:46:13.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T08:46:13.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:46:13.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e5f6f02c-ab97-413e-b2c9-6a75093bed64 2024-04-25T08:46:13.262 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 e5f6f02c-ab97-413e-b2c9-6a75093bed64 2024-04-25T08:46:13.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e5f6f02c-ab97-413e-b2c9-6a75093bed64' 2024-04-25T08:46:13.262 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:46:13.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVGCpmmc5BERAACWLPRIDioqa7NsVrkGMijg== 2024-04-25T08:46:13.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVGCpmmc5BERAACWLPRIDioqa7NsVrkGMijg=="}' 2024-04-25T08:46:13.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e5f6f02c-ab97-413e-b2c9-6a75093bed64 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:46:13.580 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:46:13.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:46:13.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVGCpmmc5BERAACWLPRIDioqa7NsVrkGMijg== --osd-uuid e5f6f02c-ab97-413e-b2c9-6a75093bed64 2024-04-25T08:46:13.623 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:13.632+0000 7f3520c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:13.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:13.656+0000 7f3520c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:13.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:13.656+0000 7f3520c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:13.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:13.656+0000 7f3520c907c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:46:13.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:13.656+0000 7f3520c907c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:46:13.650 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:13.656+0000 7f3520c907c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:46:13.689 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:13.696+0000 7f3520c907c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T08:46:19.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T08:46:19.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:46:19.394 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:46:19.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:46:19.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:46:19.941 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:46:19.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:46:19.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:46:19.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:46:19.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:46:19.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:46:19.970 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:19.976+0000 7f87f2e697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:19.979 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:19.988+0000 7f87f2e697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:19.981 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:19.988+0000 7f87f2e697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:46:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:46:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:46:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:46:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:46:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:46:20.341 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:46:20.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:20.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:20.546 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:20.552+0000 7f87f2e697c0 -1 Falling back to public interface 2024-04-25T08:46:20.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:21.786 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:46:21.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:21.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:21.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:46:21.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:21.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:22.195 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:23.050 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:23.056+0000 7f87f2e697c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:46:23.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:23.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:23.197 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:46:23.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:46:23.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:23.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:23.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:24.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:24.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:24.655 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:46:24.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:46:24.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:24.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:25.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:26.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:26.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:26.077 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:46:26.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:46:26.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:26.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:26.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:27.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:27.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:27.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:46:27.510 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:46:27.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:27.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:28.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:29.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:29.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:29.068 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:46:29.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:46:29.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:29.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:29.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:30.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:30.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:30.517 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:46:30.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:46:30.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:30.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:31.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:32.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:32.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:32.012 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:46:32.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:46:32.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:32.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:32.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:33.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:33.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:33.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:46:33.445 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:46:33.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:33.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:33.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:34.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:34.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:34.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:46:34.903 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:46:34.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:34.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:35.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:36.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:36.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:36.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:46:36.753 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:46:36.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:36.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:37.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:46:38.320 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:46:38.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:38.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:39.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:40.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:40.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:40.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:46:40.060 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:46:40.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:40.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:41.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:42.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:42.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:42.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:46:42.189 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:46:42.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:42.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:42.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:43.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:43.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:43.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:46:43.886 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:46:43.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:43.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:44.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:45.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:45.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:45.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:46:45.328 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:46:45.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:45.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:45.827 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3709614977,v1:127.0.0.1:6803/3709614977] [v2:127.0.0.1:6804/3709614977,v1:127.0.0.1:6805/3709614977] exists,up e5f6f02c-ab97-413e-b2c9-6a75093bed64 2024-04-25T08:46:45.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:46:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:46:45.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:46:45.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T08:46:45.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:46:45.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=073cdffb-769a-4663-9f7b-8f984850bb5e 2024-04-25T08:46:45.834 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 073cdffb-769a-4663-9f7b-8f984850bb5e 2024-04-25T08:46:45.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 073cdffb-769a-4663-9f7b-8f984850bb5e' 2024-04-25T08:46:45.834 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:46:45.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1GCpmEIpjMxAAemWlSZ/++viEwxB7VgHWIg== 2024-04-25T08:46:45.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1GCpmEIpjMxAAemWlSZ/++viEwxB7VgHWIg=="}' 2024-04-25T08:46:45.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 073cdffb-769a-4663-9f7b-8f984850bb5e -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:46:46.427 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:46:46.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:46:46.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1GCpmEIpjMxAAemWlSZ/++viEwxB7VgHWIg== --osd-uuid 073cdffb-769a-4663-9f7b-8f984850bb5e 2024-04-25T08:46:46.474 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:46.479+0000 7f2a0e6bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:46.476 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:46.483+0000 7f2a0e6bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:46.477 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:46.483+0000 7f2a0e6bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:46.478 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:46.483+0000 7f2a0e6bb7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:46:46.478 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:46.483+0000 7f2a0e6bb7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:46:46.478 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:46.483+0000 7f2a0e6bb7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:46:46.566 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:46.571+0000 7f2a0e6bb7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T08:46:52.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T08:46:52.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:46:52.342 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:46:52.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:46:52.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:46:52.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:46:52.905 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:46:52.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:46:52.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:46:52.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:46:52.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:46:52.928 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:52.935+0000 7fa015ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:52.931 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:52.939+0000 7fa015ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:52.932 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:52.939+0000 7fa015ecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:53.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:46:53.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:46:53.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:46:53.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:46:53.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:46:53.343 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:46:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:46:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:53.495+0000 7fa015ecc7c0 -1 Falling back to public interface 2024-04-25T08:46:53.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:54.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:54.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:54.772 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:46:54.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:46:54.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:54.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:55.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:56.153 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:46:56.159+0000 7fa015ecc7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:46:56.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:56.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:56.172 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:46:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:46:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:56.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:56.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:57.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:57.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:57.652 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:46:57.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:46:57.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:57.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:58.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:59.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:59.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:59.137 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:46:59.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:46:59.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:59.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:59.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:00.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:00.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:00.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:47:00.551 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:47:00.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:00.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:01.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:02.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:02.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:02.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:47:02.055 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:47:02.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:02.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:02.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:03.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:03.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:03.558 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:47:03.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:47:03.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:03.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:04.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:05.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:05.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:05.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:47:05.102 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:47:05.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:05.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:05.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:06.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:06.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:06.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:47:06.512 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:47:06.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:06.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:07.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:08.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:08.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:08.018 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:47:08.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:47:08.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:08.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:08.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:09.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:09.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:09.741 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:47:09.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:47:09.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:09.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:10.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:11.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:11.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:11.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:47:11.825 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:47:11.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:11.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:12.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:13.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:13.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:13.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:47:13.632 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:47:13.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:13.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:14.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:15.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:15.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:15.415 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:47:15.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:47:15.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:15.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:15.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:16.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:16.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:16.866 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:47:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:47:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:16.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:17.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:18.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:18.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:18.594 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:47:18.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:47:18.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:18.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:19.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:20.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:20.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:20.005 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:47:20.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:47:20.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:20.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:47:20.539 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1413870850,v1:127.0.0.1:6811/1413870850] [v2:127.0.0.1:6812/1413870850,v1:127.0.0.1:6813/1413870850] exists,up 073cdffb-769a-4663-9f7b-8f984850bb5e 2024-04-25T08:47:20.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:47:20.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:47:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:47:20.543 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:47:20.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:47:20.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:47:20.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T08:47:20.545 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:47:20.546 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 158029d6-c763-4e32-8bea-7175b66a64f0 2024-04-25T08:47:20.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=158029d6-c763-4e32-8bea-7175b66a64f0 2024-04-25T08:47:20.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 158029d6-c763-4e32-8bea-7175b66a64f0' 2024-04-25T08:47:20.546 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:47:20.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYGCpmFsk2IhAA8lxz7/Gns9YA9HukttcXlA== 2024-04-25T08:47:20.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYGCpmFsk2IhAA8lxz7/Gns9YA9HukttcXlA=="}' 2024-04-25T08:47:20.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 158029d6-c763-4e32-8bea-7175b66a64f0 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T08:47:21.053 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:47:21.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T08:47:21.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYGCpmFsk2IhAA8lxz7/Gns9YA9HukttcXlA== --osd-uuid 158029d6-c763-4e32-8bea-7175b66a64f0 2024-04-25T08:47:21.098 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:21.103+0000 7f3fe26b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:21.100 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:21.107+0000 7f3fe26b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:21.101 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:21.107+0000 7f3fe26b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:21.102 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:21.107+0000 7f3fe26b97c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:47:21.102 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:21.107+0000 7f3fe26b97c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:47:21.102 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:21.107+0000 7f3fe26b97c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:47:21.204 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:21.211+0000 7f3fe26b97c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T08:47:27.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T08:47:27.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:47:27.350 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:47:27.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:47:27.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:47:27.904 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:47:27.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:47:27.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:47:27.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:47:27.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:47:27.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:47:27.933 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:27.939+0000 7f4887f727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:27.950 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:27.955+0000 7f4887f727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:27.951 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:27.959+0000 7f4887f727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:28.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:47:28.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:47:28.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:47:28.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:47:28.337 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:47:28.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:47:28.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:28.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:47:28.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:28.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:28.544 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:28.551+0000 7f4887f727c0 -1 Falling back to public interface 2024-04-25T08:47:28.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:29.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:29.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:29.771 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:47:29.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:47:29.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:29.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:30.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:31.027 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:31.031+0000 7f4887f727c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:47:31.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:31.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:31.174 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:47:31.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:47:31.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:31.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:31.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:32.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:32.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:32.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:47:32.766 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:47:32.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:32.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:33.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:34.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:34.313 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:47:34.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:34.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:47:34.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:34.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:34.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:35.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:35.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:35.736 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:47:35.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:47:35.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:35.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:36.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:37.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:37.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:37.281 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:47:37.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:47:37.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:37.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:37.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:38.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:38.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:38.720 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:47:38.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:47:38.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:38.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:39.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:40.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:40.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:40.189 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:47:40.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:47:40.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:40.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:40.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:41.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:41.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:41.643 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:47:41.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:47:41.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:41.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:42.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:43.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:43.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:43.314 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:47:43.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:47:43.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:43.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:43.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:44.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:44.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:44.756 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:47:44.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:47:44.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:44.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:45.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:46.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:46.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:46.573 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:47:46.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:47:46.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:46.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:47.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:48.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:48.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:48.485 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:47:48.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:47:48.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:48.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:49.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:50.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:50.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:50.272 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:47:50.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:47:50.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:50.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:51.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:52.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:52.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:52.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:47:52.319 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:47:52.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:52.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:53.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:54.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:54.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:54.090 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:47:54.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:47:54.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:54.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:54.587 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2274800911,v1:127.0.0.1:6819/2274800911] [v2:127.0.0.1:6820/2274800911,v1:127.0.0.1:6821/2274800911] exists,up 158029d6-c763-4e32-8bea-7175b66a64f0 2024-04-25T08:47:54.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:47:54.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:47:54.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:47:54.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:47:54.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-04-25T08:47:54.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:47:54.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:47:54.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:47:54.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:47:54.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:47:54.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-04-25T08:47:54.594 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:47:54.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a071d3c-4347-46da-a2e9-b20700775586 2024-04-25T08:47:54.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1a071d3c-4347-46da-a2e9-b20700775586' 2024-04-25T08:47:54.595 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 1a071d3c-4347-46da-a2e9-b20700775586 2024-04-25T08:47:54.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:47:54.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6GCpmF3kUJRAAayj6xA2AGsoyuPX0fzeQoQ== 2024-04-25T08:47:54.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6GCpmF3kUJRAAayj6xA2AGsoyuPX0fzeQoQ=="}' 2024-04-25T08:47:54.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a071d3c-4347-46da-a2e9-b20700775586 -i td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T08:47:55.374 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:47:55.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T08:47:55.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6GCpmF3kUJRAAayj6xA2AGsoyuPX0fzeQoQ== --osd-uuid 1a071d3c-4347-46da-a2e9-b20700775586 2024-04-25T08:47:55.421 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:55.426+0000 7f1fc03e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:55.430 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:55.434+0000 7f1fc03e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:55.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:55.438+0000 7f1fc03e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:55.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:55.438+0000 7f1fc03e07c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:47:55.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:55.438+0000 7f1fc03e07c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:47:55.432 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:55.438+0000 7f1fc03e07c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:47:55.543 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:47:55.550+0000 7f1fc03e07c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-04-25T08:48:01.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-04-25T08:48:01.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:48:01.418 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T08:48:01.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:48:01.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:48:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:48:01.982 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T08:48:01.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:48:01.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:48:01.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:48:01.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:48:02.011 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:48:02.018+0000 7f512cfb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:02.013 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:48:02.018+0000 7f512cfb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:02.014 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:48:02.022+0000 7f512cfb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:02.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:48:02.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:48:02.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:48:02.418 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:48:02.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:48:02.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:48:02.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:02.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:48:02.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:02.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:02.580 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:48:02.586+0000 7f512cfb67c0 -1 Falling back to public interface 2024-04-25T08:48:02.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:03.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:03.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:03.856 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:48:03.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:48:03.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:03.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:04.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:05.072 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:48:05.078+0000 7f512cfb67c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:48:05.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:05.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:05.261 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:48:05.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:48:05.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:05.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:05.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:06.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:48:06.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:06.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:07.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:08.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:08.555 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:48:08.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:08.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:48:08.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:08.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:09.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:10.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:10.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:10.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:48:10.019 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:48:10.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:10.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:10.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:11.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:11.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:11.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:48:11.545 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:48:11.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:11.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:12.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:13.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:13.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:13.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:48:13.011 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:48:13.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:13.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:13.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:14.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:14.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:14.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:48:14.438 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:48:14.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:14.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:14.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:15.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:15.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:15.900 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:48:15.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:48:15.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:15.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:16.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:17.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:17.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:17.428 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:48:17.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:48:17.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:17.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:18.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:19.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:19.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:19.260 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:48:19.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:48:19.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:19.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:19.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:20.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:20.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:20.863 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:48:20.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:48:20.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:20.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:22.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:48:23.064 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:48:23.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:23.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:23.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:24.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:24.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:24.536 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:48:24.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:48:24.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:24.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:25.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:26.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:26.324 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:48:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:48:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:26.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:27.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:28.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:28.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:28.179 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:48:28.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:48:28.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:28.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:29.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:30.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:30.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:30.002 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:48:30.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:48:30.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:30.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:30.453 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3927716869,v1:127.0.0.1:6827/3927716869] [v2:127.0.0.1:6828/3927716869,v1:127.0.0.1:6829/3927716869] exists,up 1a071d3c-4347-46da-a2e9-b20700775586 2024-04-25T08:48:30.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:48:30.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:48:30.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:48:30.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T08:48:30.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:48:30.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:48:30.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:48:30.455 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:48:30.456 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:48:30.456 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:48:30.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:48:30.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:48:30.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:48:30.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:48:30.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:48:30.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:48:30.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:48:30.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:48:30.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:48:30.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:48:30.951 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:48:30.951 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:48:30.952 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:48:30.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:48:30.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:30.952 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:48:31.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803805 2024-04-25T08:48:31.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803805 2024-04-25T08:48:31.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803805' 2024-04-25T08:48:31.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:31.111 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:48:31.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607573 2024-04-25T08:48:31.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607573 2024-04-25T08:48:31.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803805 1-51539607573' 2024-04-25T08:48:31.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:31.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:48:31.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345933 2024-04-25T08:48:31.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345933 2024-04-25T08:48:31.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803805 1-51539607573 2-85899345933' 2024-04-25T08:48:31.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:31.390 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:48:31.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920776 2024-04-25T08:48:31.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920776 2024-04-25T08:48:31.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803805 1-51539607573 2-85899345933 3-141733920776' 2024-04-25T08:48:31.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:31.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803805 2024-04-25T08:48:31.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:31.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:48:31.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803805 2024-04-25T08:48:31.531 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:31.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803805 2024-04-25T08:48:31.532 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803805 2024-04-25T08:48:31.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2024-04-25T08:48:31.532 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:48:31.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803805 -lt 25769803805 2024-04-25T08:48:31.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:31.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607573 2024-04-25T08:48:31.912 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:31.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:48:31.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607573 2024-04-25T08:48:31.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:31.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607573 2024-04-25T08:48:31.915 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607573 2024-04-25T08:48:31.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607573' 2024-04-25T08:48:31.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:48:32.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607573 -lt 51539607573 2024-04-25T08:48:32.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:32.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345933 2024-04-25T08:48:32.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:32.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:48:32.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345933 2024-04-25T08:48:32.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:32.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345933 2024-04-25T08:48:32.397 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345933 2024-04-25T08:48:32.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345933' 2024-04-25T08:48:32.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:48:32.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345933 -lt 85899345933 2024-04-25T08:48:32.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:32.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:32.817 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920776 2024-04-25T08:48:32.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:48:32.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920776 2024-04-25T08:48:32.819 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:32.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920776 2024-04-25T08:48:32.820 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920776 2024-04-25T08:48:32.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920776' 2024-04-25T08:48:32.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:48:33.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920774 -lt 141733920776 2024-04-25T08:48:33.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:48:34.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:48:34.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:48:34.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920776 -lt 141733920776 2024-04-25T08:48:34.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:48:34.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:34.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:35.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T08:48:35.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:48:35.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:48:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:48:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:48:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:48:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:48:35.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:48:35.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-04-25T08:48:35.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:48:35.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:35.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-04-25T08:48:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' -1 2024-04-25T08:48:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:48:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=7 2024-04-25T08:48:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:48:36.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:48:36.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:48:36.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:48:36.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:48:36.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:48:36.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:48:36.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:48:36.368 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:48:36.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:48:36.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-04-25T08:48:36.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:48:36.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:36.799 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:37.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-04-25T08:48:37.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 7 2024-04-25T08:48:37.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:48:37.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:48:37.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:48:37.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:48:37.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:48:37.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:48:37.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:48:37.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:48:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:48:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T08:48:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:48:37.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T08:48:38.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:48:38.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:48:38.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:48:38.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:48:38.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:48:38.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:48:38.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:48:38.094 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:48:38.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-04-25T08:48:38.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:48:38.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:38.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-04-25T08:48:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 7 2024-04-25T08:48:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:48:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:48:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:48:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:48:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:48:39.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:48:39.009 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:48:39.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:48:39.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:48:39.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-25T08:48:39.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:48:39.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T08:48:39.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:48:39.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:48:39.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:48:39.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:48:39.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:48:39.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:48:39.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:48:39.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:48:40.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-04-25T08:48:40.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:48:40.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:40.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:40.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-04-25T08:48:40.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 7 2024-04-25T08:48:40.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:48:40.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:48:40.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:48:40.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:48:40.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:48:40.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:48:40.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:48:41.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:48:41.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:48:41.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-04-25T08:48:41.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:48:41.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T08:48:42.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:48:42.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:48:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:48:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:48:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:48:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:48:42.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:48:42.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:48:42.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:48:42.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:48:42.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:42.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:43.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T08:48:43.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:48:43.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:48:43.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:172: TEST_rados_get_with_eio: local poolname=pool-rep 2024-04-25T08:48:43.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:173: TEST_rados_get_with_eio: create_pool pool-rep 1 1 2024-04-25T08:48:43.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-04-25T08:48:43.785 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'pool-rep' already exists 2024-04-25T08:48:43.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:48:44.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:174: TEST_rados_get_with_eio: wait_for_clean 2024-04-25T08:48:44.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:48:44.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:48:44.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:48:44.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:48:44.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:48:44.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:48:44.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:48:44.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:48:44.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:48:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:48:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:48:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:48:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:48:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:48:44.929 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:48:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:48:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:48:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:48:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:48:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:48:45.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:45.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:48:45.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803809 2024-04-25T08:48:45.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803809 2024-04-25T08:48:45.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803809' 2024-04-25T08:48:45.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:45.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:48:45.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607578 2024-04-25T08:48:45.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607578 2024-04-25T08:48:45.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803809 1-51539607578' 2024-04-25T08:48:45.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:45.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:48:45.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345938 2024-04-25T08:48:45.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345938 2024-04-25T08:48:45.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803809 1-51539607578 2-85899345938' 2024-04-25T08:48:45.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:45.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:48:45.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920781 2024-04-25T08:48:45.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920781 2024-04-25T08:48:45.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803809 1-51539607578 2-85899345938 3-141733920781' 2024-04-25T08:48:45.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:45.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803809 2024-04-25T08:48:45.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:45.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:48:45.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803809 2024-04-25T08:48:45.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:45.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803809 2024-04-25T08:48:45.936 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803809 2024-04-25T08:48:45.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-04-25T08:48:45.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:48:46.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803809 -lt 25769803809 2024-04-25T08:48:46.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:46.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607578 2024-04-25T08:48:46.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:46.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:48:46.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607578 2024-04-25T08:48:46.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:46.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607578 2024-04-25T08:48:46.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607578' 2024-04-25T08:48:46.326 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607578 2024-04-25T08:48:46.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:48:46.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607576 -lt 51539607578 2024-04-25T08:48:46.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:48:47.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:48:47.748 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:48:48.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607578 -lt 51539607578 2024-04-25T08:48:48.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:48.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345938 2024-04-25T08:48:48.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:48.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:48:48.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:48.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345938 2024-04-25T08:48:48.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345938 2024-04-25T08:48:48.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345938' 2024-04-25T08:48:48.178 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345938 2024-04-25T08:48:48.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:48:48.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345938 -lt 85899345938 2024-04-25T08:48:48.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:48.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:48.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920781 2024-04-25T08:48:48.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:48:48.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920781 2024-04-25T08:48:48.653 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920781 2024-04-25T08:48:48.654 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920781 2024-04-25T08:48:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920781' 2024-04-25T08:48:48.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:48:49.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920781 -lt 141733920781 2024-04-25T08:48:49.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:48:49.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:49.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:49.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:48:49.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:48:49.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:48:49.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:48:49.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:48:49.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:48:49.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:48:49.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:48:50.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:48:50.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:48:50.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:50.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:175: TEST_rados_get_with_eio: rados_get_data eio td/osd-rep-recov-eio.sh 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:111: rados_get_data: local inject=eio 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:112: rados_get_data: shift 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:113: rados_get_data: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:115: rados_get_data: local poolname=pool-rep 2024-04-25T08:48:50.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:116: rados_get_data: local objname=obj-eio-389694 2024-04-25T08:48:50.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: get_pg pool-rep obj-eio-389694 2024-04-25T08:48:50.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-04-25T08:48:50.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-eio-389694 2024-04-25T08:48:50.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:48:50.613 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:48:51.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: local pgid=2.0 2024-04-25T08:48:51.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:119: rados_get_data: rados_put td/osd-rep-recov-eio.sh pool-rep obj-eio-389694 2024-04-25T08:48:51.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:48:51.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:48:51.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-eio-389694 2024-04-25T08:48:51.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:48:51.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:48:51.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:48:51.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:48:51.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:48:51.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:48:51.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:48:51.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:48:51.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-eio-389694 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:120: rados_get_data: inject_eio rep data pool-rep obj-eio-389694 td/osd-rep-recov-eio.sh 0 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-389694 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:48:51.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:48:51.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-389694 2024-04-25T08:48:51.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:48:51.258 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:48:51.258 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:48:51.258 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:48:51.676 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:48:51.676 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:48:51.676 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:48:51.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:48:51.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:48:51.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:48:51.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:48:51.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:48:51.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:48:51.677 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:48:51.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:48:51.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:48:51.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:48:51.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:48:51.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:48:51.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:48:51.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:48:51.679 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:48:51.679 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:48:51.679 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:48:51.680 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:48:51.680 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:51.680 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:48:51.680 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:48:51.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:48:51.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:48:51.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:48:51.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:48:51.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:48:51.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:48:51.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:48:51.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:48:51.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:51.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:48:51.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:48:51.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:48:51.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-eio-389694 2024-04-25T08:48:51.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:121: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-389694 2024-04-25T08:48:51.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:48:51.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:48:51.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-389694 2024-04-25T08:48:51.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:48:51.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:48:51.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:48:51.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-389694 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:48:51.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:48:52.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:48:52.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:123: rados_get_data: wait_for_clean 2024-04-25T08:48:52.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:48:52.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:48:52.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:48:52.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:48:52.006 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:48:52.007 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:48:52.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:48:52.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:48:52.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:48:52.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:48:52.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:48:52.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:48:52.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:48:52.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:48:52.107 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:48:52.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:48:52.518 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:48:52.518 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:48:52.518 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:48:52.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:48:52.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:52.519 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:48:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803813 2024-04-25T08:48:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803813 2024-04-25T08:48:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813' 2024-04-25T08:48:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:52.711 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:48:52.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607581 2024-04-25T08:48:52.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607581 2024-04-25T08:48:52.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-51539607581' 2024-04-25T08:48:52.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:52.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:48:53.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345941 2024-04-25T08:48:53.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345941 2024-04-25T08:48:53.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-51539607581 2-85899345941' 2024-04-25T08:48:53.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:53.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:48:53.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920784 2024-04-25T08:48:53.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920784 2024-04-25T08:48:53.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-51539607581 2-85899345941 3-141733920784' 2024-04-25T08:48:53.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:53.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803813 2024-04-25T08:48:53.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:53.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:48:53.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803813 2024-04-25T08:48:53.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:53.161 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803813 2024-04-25T08:48:53.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803813 2024-04-25T08:48:53.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-04-25T08:48:53.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:48:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803811 -lt 25769803813 2024-04-25T08:48:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:48:54.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:48:54.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:48:54.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803813 -lt 25769803813 2024-04-25T08:48:54.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:54.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607581 2024-04-25T08:48:54.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:54.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:48:54.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607581 2024-04-25T08:48:54.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:54.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607581 2024-04-25T08:48:54.984 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607581 2024-04-25T08:48:54.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607581' 2024-04-25T08:48:54.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:48:55.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607581 -lt 51539607581 2024-04-25T08:48:55.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:55.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:55.399 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345941 2024-04-25T08:48:55.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:48:55.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345941 2024-04-25T08:48:55.401 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:55.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345941 2024-04-25T08:48:55.403 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345941 2024-04-25T08:48:55.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345941' 2024-04-25T08:48:55.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:48:55.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345942 -lt 85899345941 2024-04-25T08:48:55.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:55.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920784 2024-04-25T08:48:55.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:55.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:48:55.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920784 2024-04-25T08:48:55.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:55.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920784 2024-04-25T08:48:55.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920784' 2024-04-25T08:48:55.830 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920784 2024-04-25T08:48:55.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:48:56.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920784 -lt 141733920784 2024-04-25T08:48:56.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:48:56.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:56.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:56.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:48:56.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:48:56.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:48:56.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:48:56.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:48:56.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:48:56.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:48:56.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:48:57.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:48:57.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:48:57.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:48:57.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:48:57.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:48:57.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:48:57.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:48:57.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: ceph pg 2.0 query 2024-04-25T08:48:57.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T08:48:57.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: COUNT=1 2024-04-25T08:48:57.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:125: rados_get_data: test 1 = 1 2024-04-25T08:48:57.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:126: rados_get_data: flush_pg_stats 2024-04-25T08:48:57.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:48:57.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:48:58.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:48:58.351 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:48:58.351 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:48:58.351 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:48:58.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:48:58.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:58.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:48:58.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803816 2024-04-25T08:48:58.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803816 2024-04-25T08:48:58.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803816' 2024-04-25T08:48:58.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:58.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:48:58.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607584 2024-04-25T08:48:58.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607584 2024-04-25T08:48:58.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803816 1-51539607584' 2024-04-25T08:48:58.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:58.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:48:58.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345944 2024-04-25T08:48:58.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345944 2024-04-25T08:48:58.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803816 1-51539607584 2-85899345944' 2024-04-25T08:48:58.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:48:58.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:48:58.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920787 2024-04-25T08:48:58.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920787 2024-04-25T08:48:58.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803816 1-51539607584 2-85899345944 3-141733920787' 2024-04-25T08:48:58.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:48:58.896 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803816 2024-04-25T08:48:58.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:48:58.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:48:58.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803816 2024-04-25T08:48:58.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:48:58.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803816 2024-04-25T08:48:58.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-04-25T08:48:58.900 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803816 2024-04-25T08:48:58.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:48:59.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803814 -lt 25769803816 2024-04-25T08:48:59.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:49:00.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:49:00.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:00.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803816 -lt 25769803816 2024-04-25T08:49:00.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:00.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607584 2024-04-25T08:49:00.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:49:00.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607584 2024-04-25T08:49:00.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:00.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607584 2024-04-25T08:49:00.707 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607584 2024-04-25T08:49:00.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607584' 2024-04-25T08:49:00.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:49:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607584 -lt 51539607584 2024-04-25T08:49:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:01.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345944 2024-04-25T08:49:01.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:01.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:49:01.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345944 2024-04-25T08:49:01.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:01.192 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345944 2024-04-25T08:49:01.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345944 2024-04-25T08:49:01.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345944' 2024-04-25T08:49:01.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:49:01.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345944 -lt 85899345944 2024-04-25T08:49:01.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:01.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920787 2024-04-25T08:49:01.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:49:01.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:01.662 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920787 2024-04-25T08:49:01.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920787 2024-04-25T08:49:01.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920787' 2024-04-25T08:49:01.664 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920787 2024-04-25T08:49:01.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:49:02.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920788 -lt 141733920787 2024-04-25T08:49:02.053 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T08:49:02.054 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T08:49:02.452 INFO:tasks.workunit.client.0.smithi088.stderr:dumped all 2024-04-25T08:49:02.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: COUNT=1 2024-04-25T08:49:02.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:128: rados_get_data: test 1 = 1 2024-04-25T08:49:02.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: get_osds pool-rep obj-eio-389694 2024-04-25T08:49:02.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:49:02.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:49:02.469 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:49:02.470 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:49:02.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:49:02.868 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:02.868 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:49:02.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:49:02.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: object_osds=('3' '1' '0') 2024-04-25T08:49:02.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: local object_osds 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:131: rados_get_data: local primary=3 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:132: rados_get_data: local bad_peer=1 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:133: rados_get_data: inject_eio rep data pool-rep obj-eio-389694 td/osd-rep-recov-eio.sh 0 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-389694 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:02.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:49:02.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:49:02.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:49:02.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-389694 2024-04-25T08:49:02.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:49:02.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:49:02.870 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:49:02.870 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:49:03.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:49:03.284 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:49:03.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:49:03.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:49:03.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:49:03.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:49:03.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:49:03.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:49:03.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:49:03.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:49:03.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:49:03.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:49:03.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-eio-389694 2024-04-25T08:49:03.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:134: rados_get_data: inject_eio rep data pool-rep obj-eio-389694 td/osd-rep-recov-eio.sh 1 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-389694 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:49:03.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:49:03.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:49:03.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-389694 2024-04-25T08:49:03.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:49:03.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:49:03.503 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:49:03.503 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:49:03.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:49:03.947 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:49:03.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:49:03.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:49:03.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:49:03.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:49:03.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:49:03.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:49:03.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:49:03.950 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:49:03.950 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:49:03.950 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:49:03.950 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:03.950 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:03.950 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:03.951 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:49:03.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:49:04.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:49:04.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:49:04.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:49:04.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:49:04.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:49:04.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:49:04.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:04.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:04.070 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:04.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:49:04.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:49:04.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-eio-389694 2024-04-25T08:49:04.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:136: rados_get_data: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:49:04.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:49:04.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:49:04.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:49:04.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:49:04.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:49:04.192 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:49:04.192 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:49:04.192 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:49:04.192 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:04.192 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:04.192 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:04.192 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:49:04.192 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:137: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-389694 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-389694 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:49:04.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-389694 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:49:04.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:49:04.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:49:04.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:140: rados_get_data: wait_for_clean 2024-04-25T08:49:04.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:49:04.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:49:04.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:49:04.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:49:04.339 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:49:04.339 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:49:04.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:49:04.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:49:04.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:49:04.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:49:04.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:49:04.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:49:04.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:49:04.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:49:04.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:49:04.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:49:04.846 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:04.846 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:49:04.846 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:49:04.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:49:04.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:04.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:49:05.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803819 2024-04-25T08:49:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803819 2024-04-25T08:49:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803819' 2024-04-25T08:49:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:05.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:49:05.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607588 2024-04-25T08:49:05.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607588 2024-04-25T08:49:05.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803819 1-51539607588' 2024-04-25T08:49:05.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:05.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:49:05.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345948 2024-04-25T08:49:05.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345948 2024-04-25T08:49:05.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803819 1-51539607588 2-85899345948' 2024-04-25T08:49:05.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:05.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:49:05.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920790 2024-04-25T08:49:05.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920790 2024-04-25T08:49:05.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803819 1-51539607588 2-85899345948 3-141733920790' 2024-04-25T08:49:05.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:05.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803819 2024-04-25T08:49:05.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:05.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:49:05.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803819 2024-04-25T08:49:05.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:05.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803819 2024-04-25T08:49:05.481 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803819 2024-04-25T08:49:05.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803819' 2024-04-25T08:49:05.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:05.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803819 -lt 25769803819 2024-04-25T08:49:05.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:05.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607588 2024-04-25T08:49:05.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:05.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:49:05.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607588 2024-04-25T08:49:05.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:05.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607588 2024-04-25T08:49:05.888 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607588 2024-04-25T08:49:05.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607588' 2024-04-25T08:49:05.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:49:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607588 -lt 51539607588 2024-04-25T08:49:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:06.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345948 2024-04-25T08:49:06.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:06.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:49:06.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345948 2024-04-25T08:49:06.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:06.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345948 2024-04-25T08:49:06.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345948' 2024-04-25T08:49:06.308 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345948 2024-04-25T08:49:06.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:49:06.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345948 -lt 85899345948 2024-04-25T08:49:06.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:06.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920790 2024-04-25T08:49:06.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:06.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:49:06.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920790 2024-04-25T08:49:06.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:06.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920790 2024-04-25T08:49:06.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920790' 2024-04-25T08:49:06.739 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920790 2024-04-25T08:49:06.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:49:07.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920790 -lt 141733920790 2024-04-25T08:49:07.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:49:07.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:07.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:07.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:49:07.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:07.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:07.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:07.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:07.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:07.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:07.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:08.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:49:08.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:08.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:08.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:49:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:49:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:49:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:142: rados_get_data: inject_eio rep data pool-rep obj-eio-389694 td/osd-rep-recov-eio.sh 0 2024-04-25T08:49:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:49:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:49:08.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-389694 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:49:08.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:49:08.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-389694 2024-04-25T08:49:08.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:49:08.647 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:49:08.647 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:49:08.647 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:49:09.103 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:49:09.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:49:09.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:49:09.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:49:09.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:49:09.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:49:09.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:49:09.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:49:09.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:49:09.108 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:49:09.108 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:49:09.108 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:49:09.109 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:49:09.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:49:09.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:49:09.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:49:09.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:49:09.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:49:09.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:49:09.227 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:09.227 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:09.227 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:09.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:49:09.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:49:09.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-eio-389694 2024-04-25T08:49:09.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:143: rados_get_data: inject_eio rep data pool-rep obj-eio-389694 td/osd-rep-recov-eio.sh 2 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-389694 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=2 2024-04-25T08:49:09.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:49:09.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-389694 2024-04-25T08:49:09.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:49:09.347 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:49:09.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:49:09.347 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:49:09.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:49:09.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-04-25T08:49:09.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:49:09.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-25T08:49:09.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:49:09.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-25T08:49:09.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:49:09.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:49:09.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:49:09.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-25T08:49:09.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:49:09.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:49:09.759 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:09.760 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:09.760 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:09.760 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.0.asok 2024-04-25T08:49:09.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:49:09.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:49:09.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:49:09.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:49:09.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-04-25T08:49:09.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:49:09.869 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:49:09.870 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:09.870 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:09.870 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:09.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.0.asok 2024-04-25T08:49:09.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:49:09.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.0.asok injectdataerr pool-rep obj-eio-389694 2024-04-25T08:49:09.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:144: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-389694 2024-04-25T08:49:09.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:09.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:49:09.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-389694 2024-04-25T08:49:09.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:49:09.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:49:09.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:49:09.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-389694 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:49:10.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:49:10.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:49:10.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:146: rados_get_data: wait_for_clean 2024-04-25T08:49:10.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:49:10.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:49:10.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:49:10.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:49:10.011 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:49:10.011 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:49:10.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:49:10.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:49:10.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:49:10.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:49:10.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:49:10.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:49:10.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:49:10.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:49:10.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:49:10.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:49:10.495 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:10.495 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:49:10.495 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:49:10.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:49:10.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:10.495 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:49:10.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803823 2024-04-25T08:49:10.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803823 2024-04-25T08:49:10.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823' 2024-04-25T08:49:10.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:10.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:49:10.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607591 2024-04-25T08:49:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607591 2024-04-25T08:49:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-51539607591' 2024-04-25T08:49:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:10.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:49:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345951 2024-04-25T08:49:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345951 2024-04-25T08:49:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-85899345951' 2024-04-25T08:49:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:10.945 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:49:11.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920794 2024-04-25T08:49:11.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920794 2024-04-25T08:49:11.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-51539607591 2-85899345951 3-141733920794' 2024-04-25T08:49:11.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:11.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803823 2024-04-25T08:49:11.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:11.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:49:11.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:11.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803823 2024-04-25T08:49:11.084 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803823 2024-04-25T08:49:11.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803823 2024-04-25T08:49:11.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-04-25T08:49:11.084 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:11.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803820 -lt 25769803823 2024-04-25T08:49:11.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:49:12.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:49:12.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:12.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803823 -lt 25769803823 2024-04-25T08:49:12.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:12.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607591 2024-04-25T08:49:12.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:12.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:49:12.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607591 2024-04-25T08:49:12.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:12.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607591 2024-04-25T08:49:12.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607591' 2024-04-25T08:49:12.885 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607591 2024-04-25T08:49:12.885 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:49:13.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607591 -lt 51539607591 2024-04-25T08:49:13.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:13.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:13.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345951 2024-04-25T08:49:13.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:49:13.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345951 2024-04-25T08:49:13.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:13.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345951 2024-04-25T08:49:13.355 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345951 2024-04-25T08:49:13.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345951' 2024-04-25T08:49:13.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:49:13.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345951 -lt 85899345951 2024-04-25T08:49:13.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:13.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920794 2024-04-25T08:49:13.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:13.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:49:13.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920794 2024-04-25T08:49:13.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:13.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920794 2024-04-25T08:49:13.789 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920794 2024-04-25T08:49:13.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920794' 2024-04-25T08:49:13.790 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:49:14.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920794 -lt 141733920794 2024-04-25T08:49:14.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:49:14.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:14.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:14.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:49:14.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:14.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:14.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:14.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:14.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:14.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:14.899 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:15.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:49:15.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:15.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:15.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:15.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:49:15.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:49:15.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:49:15.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: ceph pg 2.0 query 2024-04-25T08:49:15.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T08:49:16.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: COUNT=3 2024-04-25T08:49:16.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:148: rados_get_data: test 3 = 3 2024-04-25T08:49:16.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:149: rados_get_data: flush_pg_stats 2024-04-25T08:49:16.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:49:16.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:49:16.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:49:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:49:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:49:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:49:16.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:16.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:49:16.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803826 2024-04-25T08:49:16.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803826 2024-04-25T08:49:16.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826' 2024-04-25T08:49:16.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:16.641 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:49:16.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607594 2024-04-25T08:49:16.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607594 2024-04-25T08:49:16.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-51539607594' 2024-04-25T08:49:16.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:16.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:49:16.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345954 2024-04-25T08:49:16.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345954 2024-04-25T08:49:16.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-51539607594 2-85899345954' 2024-04-25T08:49:16.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:16.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:49:17.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920797 2024-04-25T08:49:17.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920797 2024-04-25T08:49:17.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-51539607594 2-85899345954 3-141733920797' 2024-04-25T08:49:17.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:17.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803826 2024-04-25T08:49:17.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:49:17.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803826 2024-04-25T08:49:17.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:17.076 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803826 2024-04-25T08:49:17.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803826 2024-04-25T08:49:17.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-04-25T08:49:17.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:17.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803823 -lt 25769803826 2024-04-25T08:49:17.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:49:18.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:49:18.464 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:18.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803826 -lt 25769803826 2024-04-25T08:49:18.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:18.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:18.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607594 2024-04-25T08:49:18.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:49:18.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607594 2024-04-25T08:49:18.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:18.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607594 2024-04-25T08:49:18.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607594' 2024-04-25T08:49:18.883 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607594 2024-04-25T08:49:18.883 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:49:19.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607594 -lt 51539607594 2024-04-25T08:49:19.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:19.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345954 2024-04-25T08:49:19.275 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:19.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:49:19.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345954 2024-04-25T08:49:19.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:19.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345954 2024-04-25T08:49:19.278 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345954 2024-04-25T08:49:19.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345954' 2024-04-25T08:49:19.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:49:19.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345954 -lt 85899345954 2024-04-25T08:49:19.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:19.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920797 2024-04-25T08:49:19.719 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:19.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:49:19.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920797 2024-04-25T08:49:19.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:19.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920797 2024-04-25T08:49:19.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920797' 2024-04-25T08:49:19.723 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920797 2024-04-25T08:49:19.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:49:20.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920797 -lt 141733920797 2024-04-25T08:49:20.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T08:49:20.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T08:49:20.559 INFO:tasks.workunit.client.0.smithi088.stderr:dumped all 2024-04-25T08:49:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: COUNT=4 2024-04-25T08:49:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:151: rados_get_data: test 4 = 4 2024-04-25T08:49:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:153: rados_get_data: inject_eio rep data pool-rep obj-eio-389694 td/osd-rep-recov-eio.sh 0 2024-04-25T08:49:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:49:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:49:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:49:20.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-389694 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-389694 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:49:20.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:49:20.578 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:49:20.578 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:49:21.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:49:21.019 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:21.019 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:49:21.019 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:49:21.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:49:21.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:49:21.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:49:21.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:49:21.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:49:21.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:49:21.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:49:21.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:49:21.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:49:21.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:49:21.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:49:21.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:49:21.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:49:21.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:49:21.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:49:21.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:49:21.021 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:21.022 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:21.022 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:21.022 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:49:21.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:49:21.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:49:21.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:49:21.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:49:21.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:49:21.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:49:21.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:49:21.136 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:21.136 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:21.136 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:21.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:49:21.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:49:21.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-eio-389694 2024-04-25T08:49:21.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:154: rados_get_data: inject_eio rep data pool-rep obj-eio-389694 td/osd-rep-recov-eio.sh 1 2024-04-25T08:49:21.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:49:21.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:49:21.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:49:21.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:49:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:49:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:49:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-389694 2024-04-25T08:49:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:49:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:49:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:49:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:49:21.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-389694 2024-04-25T08:49:21.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:49:21.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:49:21.255 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:49:21.255 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:49:21.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:49:21.687 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:21.687 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:49:21.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:49:21.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:49:21.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:49:21.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:49:21.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:49:21.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:49:21.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:49:21.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:49:21.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:49:21.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:49:21.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:49:21.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:49:21.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:49:21.691 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:49:21.692 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:49:21.692 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:49:21.692 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:49:21.692 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:21.692 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:21.692 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:21.692 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:49:21.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:49:21.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:49:21.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:49:21.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:49:21.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:49:21.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:49:21.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:49:21.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:21.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:21.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:21.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:49:21.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:49:21.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-eio-389694 2024-04-25T08:49:21.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:155: rados_get_data: inject_eio rep data pool-rep obj-eio-389694 td/osd-rep-recov-eio.sh 2 2024-04-25T08:49:21.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:49:21.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:49:21.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:49:21.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:49:21.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:49:21.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:49:21.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-389694 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=2 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-389694 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-389694 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:49:21.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-389694 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:49:22.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:49:22.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-04-25T08:49:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:49:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-25T08:49:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:49:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-25T08:49:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:49:22.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:49:22.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:49:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-25T08:49:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:49:22.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:49:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:22.341 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.0.asok 2024-04-25T08:49:22.341 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:49:22.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:49:22.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:49:22.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.0.asok 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:49:22.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.0.asok injectdataerr pool-rep obj-eio-389694 2024-04-25T08:49:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:156: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-389694 hang 2024-04-25T08:49:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:49:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:49:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-389694 2024-04-25T08:49:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=hang 2024-04-25T08:49:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' hang = fail ']' 2024-04-25T08:49:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' hang = hang ']' 2024-04-25T08:49:22.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:98: rados_get: timeout 5 rados --pool pool-rep get obj-eio-389694 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:49:27.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:99: rados_get: test 124 = 124 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:100: rados_get: return 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:158: rados_get_data: wait_for_clean 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:49:27.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:49:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:49:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:49:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:49:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:49:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:49:27.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:49:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:49:28.092 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:49:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:49:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:49:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:49:28.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:28.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:49:28.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803830 2024-04-25T08:49:28.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803830 2024-04-25T08:49:28.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830' 2024-04-25T08:49:28.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:28.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:49:28.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607598 2024-04-25T08:49:28.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607598 2024-04-25T08:49:28.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607598' 2024-04-25T08:49:28.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:28.389 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:49:28.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345958 2024-04-25T08:49:28.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345958 2024-04-25T08:49:28.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607598 2-85899345958' 2024-04-25T08:49:28.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:28.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:49:28.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920801 2024-04-25T08:49:28.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920801 2024-04-25T08:49:28.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607598 2-85899345958 3-141733920801' 2024-04-25T08:49:28.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:28.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803830 2024-04-25T08:49:28.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:28.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:49:28.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803830 2024-04-25T08:49:28.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:28.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803830 2024-04-25T08:49:28.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-04-25T08:49:28.687 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803830 2024-04-25T08:49:28.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:29.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803828 -lt 25769803830 2024-04-25T08:49:29.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:49:30.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:49:30.070 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803830 -lt 25769803830 2024-04-25T08:49:30.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:30.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:30.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607598 2024-04-25T08:49:30.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:49:30.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:30.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607598 2024-04-25T08:49:30.484 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607598 2024-04-25T08:49:30.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607598 2024-04-25T08:49:30.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607598' 2024-04-25T08:49:30.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:49:30.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607598 -lt 51539607598 2024-04-25T08:49:30.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:30.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345958 2024-04-25T08:49:30.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:30.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:49:30.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345958 2024-04-25T08:49:30.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:30.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345958 2024-04-25T08:49:30.918 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345958 2024-04-25T08:49:30.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345958' 2024-04-25T08:49:30.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:49:31.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345958 -lt 85899345958 2024-04-25T08:49:31.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:31.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920801 2024-04-25T08:49:31.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:31.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:49:31.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920801 2024-04-25T08:49:31.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920801 2024-04-25T08:49:31.325 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920801 2024-04-25T08:49:31.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920801' 2024-04-25T08:49:31.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:49:31.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920802 -lt 141733920801 2024-04-25T08:49:31.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:49:31.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:31.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:32.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:49:32.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:32.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:32.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:32.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:32.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:32.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:32.264 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:32.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:49:32.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:32.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:32.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:33.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:49:33.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-04-25T08:49:33.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:49:33.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-04-25T08:49:33.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:49:33.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:49:33.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:49:33.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:33.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:33.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:33.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:33.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:33.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:33.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:33.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:49:33.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:33.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:33.754 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:34.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:49:34.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:49:34.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:49:34.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:49:34.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:49:34.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:49:34.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:49:34.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:49:34.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:49:34.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:49:34.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:49:34.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T08:49:34.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:49:34.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T08:49:35.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:49:35.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:35.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:35.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:35.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:35.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:35.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:35.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:35.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:49:35.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:35.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:35.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:35.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:49:35.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:49:35.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:49:35.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:49:35.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:49:35.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:49:35.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:49:35.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:49:35.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:49:36.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:49:36.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:49:36.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-25T08:49:36.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:49:36.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T08:49:36.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:49:36.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:36.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:36.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:36.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:36.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:36.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:36.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:37.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:49:37.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:37.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:37.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:49:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:49:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:49:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:49:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:49:37.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:49:37.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:49:37.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:49:37.836 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:49:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:49:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:49:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-04-25T08:49:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:49:38.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T08:49:39.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:49:39.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:39.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:39.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:39.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:39.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:39.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:39.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:39.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:49:39.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:39.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:39.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:49:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:49:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:49:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:49:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:49:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:49:40.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:49:40.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:49:40.120 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:49:40.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:49:40.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:49:40.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 13 )) 2024-04-25T08:49:40.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:49:40.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T08:49:42.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:49:42.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:42.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:42.268 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:42.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:49:42.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:42.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:42.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:43.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:49:43.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:49:43.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:49:43.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:49:43.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:49:43.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:49:43.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:49:43.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:49:43.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:49:43.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:49:43.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:49:43.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 13 )) 2024-04-25T08:49:43.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:49:43.766 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T08:49:46.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:49:46.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:46.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:46.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:46.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:46.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:46.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:46.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:47.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:49:47.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:47.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:47.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:47.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:49:47.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:49:47.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:49:47.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:49:47.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:49:47.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:49:47.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:49:47.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:49:47.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:49:48.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:49:48.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:49:48.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 13 )) 2024-04-25T08:49:48.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:49:48.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T08:49:54.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:49:54.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:54.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:54.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:54.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:54.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:54.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:54.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:55.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:49:55.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:55.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:55.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:55.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:49:56.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:49:56.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:49:56.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 13 )) 2024-04-25T08:49:56.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:49:56.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 12.8 2024-04-25T08:50:09.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:09.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:09.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:09.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:09.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:09.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:09.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:09.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:09.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:50:09.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:09.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:09.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:10.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:50:10.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:50:10.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:10.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:10.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:10.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:10.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:10.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:10.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:10.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:50:10.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:50:10.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 13 )) 2024-04-25T08:50:10.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:10.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T08:50:25.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:25.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:25.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:25.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:25.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:25.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:25.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:25.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:26.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:50:26.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:26.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:26.162 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:26.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:50:26.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:50:26.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:26.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:26.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:26.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:26.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:26.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:26.723 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:27.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:50:27.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:50:27.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 9 >= 13 )) 2024-04-25T08:50:27.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:27.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T08:50:42.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:42.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:42.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:42.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:42.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:42.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:50:42.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:42.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:42.740 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:43.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:50:43.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:50:43.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:43.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:43.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:43.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:43.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:43.269 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:43.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:50:43.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:50:43.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 10 >= 13 )) 2024-04-25T08:50:43.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:43.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T08:50:58.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:58.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:58.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:58.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:58.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:58.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:58.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:58.816 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:59.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:50:59.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:59.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:59.231 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:59.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:50:59.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:50:59.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:59.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:59.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:59.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:59.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:59.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:59.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:51:00.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:51:00.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:51:00.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 11 >= 13 )) 2024-04-25T08:51:00.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:51:00.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T08:51:15.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:51:15.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:51:15.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:51:15.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:51:15.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:51:15.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:51:15.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:51:15.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:51:15.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:51:15.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:51:15.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:51:15.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:51:16.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:51:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:51:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:51:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:51:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:51:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:51:16.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:51:16.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:51:16.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:51:16.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:51:16.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:51:16.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 12 >= 13 )) 2024-04-25T08:51:16.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:51:16.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 4.5 2024-04-25T08:51:21.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:51:21.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:51:21.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:51:21.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:51:21.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:51:21.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:51:21.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:51:21.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:51:21.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:51:21.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:51:21.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:51:21.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:51:22.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:51:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:51:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:51:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:51:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:51:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:51:22.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:51:22.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:51:22.359 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:51:22.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:51:22.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:51:22.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 13 >= 13 )) 2024-04-25T08:51:22.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T08:51:23.457 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T08:51:23.457 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_fingerprint": "eb999b16-42a2-470e-baf5-f0dbb8089415", 2024-04-25T08:51:23.457 INFO:tasks.workunit.client.0.smithi088.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T08:51:23.457 INFO:tasks.workunit.client.0.smithi088.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T08:51:23.457 INFO:tasks.workunit.client.0.smithi088.stdout: "timestamp": "2024-04-25T08:51:23.465501+0000", 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "tag": "", 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "health": { 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "status": "HEALTH_ERR", 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "checks": { 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "OBJECT_UNFOUND": { 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1/1 objects unfound (100.000%)", 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 2.0 has 1 unfound objects" 2024-04-25T08:51:23.458 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DAMAGED": { 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_ERR", 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Possible data damage: 1 pg recovery_unfound", 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T08:51:23.459 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "PG_DEGRADED": { 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "Degraded data redundancy: 3/3 objects degraded (100.000%), 1 pg degraded", 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.460 INFO:tasks.workunit.client.0.smithi088.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "2 pool(s) do not have an application enabled", 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 2 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [ 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'foo'" 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "application not enabled on pool 'pool-rep'" 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T08:51:23.461 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: "SLOW_OPS": { 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: "severity": "HEALTH_WARN", 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: "summary": { 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: "message": "1 slow ops, oldest one blocked for 117 sec, osd.3 has slow ops", 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: "count": 1 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: "detail": [], 2024-04-25T08:51:23.462 INFO:tasks.workunit.client.0.smithi088.stdout: "muted": false 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "mutes": [] 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_first_committed": 1, 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap_last_committed": 1, 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "monmap": { 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "216e48ad-ddcc-4ac6-8ab9-7bf658b33b37", 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T08:46:11.151900Z", 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T08:46:11.151900Z", 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release": 19, 2024-04-25T08:51:23.463 INFO:tasks.workunit.client.0.smithi088.stdout: "min_mon_release_name": "squid", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "election_strategy": 1, 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "disallowed_leaders: ": "", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": false, 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "tiebreaker_mon": "", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_ranks: ": "", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "features": { 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "persistent": [ 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "kraken", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "luminous", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "mimic", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap-prune", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "nautilus", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "octopus", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "pacific", 2024-04-25T08:51:23.464 INFO:tasks.workunit.client.0.smithi088.stdout: "elector-pinging", 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "quincy", 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "reef", 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "squid" 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "optional": [] 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "mons": [ 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "rank": 0, 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "a", 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.465 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7140", 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 0 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:7140/0", 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:7140/0", 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: "priority": 0, 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 0, 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_location": "{}" 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: "quorum": [ 2024-04-25T08:51:23.466 INFO:tasks.workunit.client.0.smithi088.stdout: 0 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap": { 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 39, 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "fsid": "216e48ad-ddcc-4ac6-8ab9-7bf658b33b37", 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "created": "2024-04-25T08:46:12.127952+0000", 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T08:48:45.239743+0000", 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "last_up_change": "2024-04-25T08:48:29.450466+0000", 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "last_in_change": "2024-04-25T08:47:54.912479+0000", 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_num": 5799936, 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_set": [ 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "pglog_hardlimit", 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snapdirs", 2024-04-25T08:51:23.467 INFO:tasks.workunit.client.0.smithi088.stdout: "recovery_deletes", 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "sortbitwise" 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_version": 9, 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_snapshot": "", 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_max": 2, 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "max_osd": 4, 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "require_min_compat_client": "luminous", 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "min_compat_client": "luminous", 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "require_osd_release": "squid", 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "allow_crimson": false, 2024-04-25T08:51:23.468 INFO:tasks.workunit.client.0.smithi088.stdout: "pools": [ 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "foo", 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T08:46:13.147072+0000", 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 1, 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool", 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T08:51:23.469 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 8, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 8, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 8, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 8, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 8, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T08:51:23.470 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "8", 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 0, 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 0, 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T08:51:23.471 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T08:51:23.472 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T08:51:23.473 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {}, 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 1.5, 2024-04-25T08:51:23.474 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 1.5, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 1, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 1.5, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 1.5, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 2, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_name": "pool-rep", 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "create_time": "2024-04-25T08:48:43.549675+0000", 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "flags": 1, 2024-04-25T08:51:23.475 INFO:tasks.workunit.client.0.smithi088.stdout: "flags_names": "hashpspool", 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "size": 3, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "min_size": 2, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_rule": 0, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_count": 0, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_target": 0, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "object_hash": 2, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_autoscale_mode": "off", 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num": 1, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num": 1, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_placement_num_target": 1, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_target": 1, 2024-04-25T08:51:23.476 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_num_pending": 1, 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pg_merge_meta": { 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "source_pgid": "0.0", 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "ready_epoch": 0, 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_started": 0, 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": 0, 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "source_version": "0'0", 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "target_version": "0'0" 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "last_change": "39", 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend": "0", 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "auid": 0, 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_mode": "selfmanaged", 2024-04-25T08:51:23.477 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_seq": 0, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_epoch": 0, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_snaps": [], 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps": "[]", 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_bytes": 0, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "quota_max_objects": 0, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "tiers": [], 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "tier_of": -1, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "read_tier": -1, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "write_tier": -1, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_mode": "none", 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_bytes": 0, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "target_max_objects": 0, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T08:51:23.478 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_flush_age": 0, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "cache_min_evict_age": 0, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profile": "", 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_params": { 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "none" 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_period": 0, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_count": 0, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "use_gmt_hitset": true, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "min_read_recency_for_promote": 0, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "min_write_recency_for_promote": 0, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "hit_set_search_last_n": 0, 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "grade_table": [], 2024-04-25T08:51:23.479 INFO:tasks.workunit.client.0.smithi088.stdout: "stripe_width": 0, 2024-04-25T08:51:23.480 INFO:tasks.workunit.client.0.smithi088.stdout: "expected_num_objects": 0, 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "fast_read": false, 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "options": {}, 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "application_metadata": {}, 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "read_balance": { 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "score_type": "Fair distribution", 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "score_acting": 4, 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "score_stable": 4, 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_score": 0.75, 2024-04-25T08:51:23.481 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_acting": 3, 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: "raw_score_stable": 3, 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity_weighted": 1, 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity": 1, 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: "average_primary_affinity_weighted": 1 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: "osds": [ 2024-04-25T08:51:23.482 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "e5f6f02c-ab97-413e-b2c9-6a75093bed64", 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 6, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 35, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 0, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T08:51:23.483 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6802", 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3709614977 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6803", 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3709614977 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.484 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6804", 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3709614977 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6805", 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3709614977 2024-04-25T08:51:23.485 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6808", 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3709614977 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.486 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6809", 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3709614977 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6806", 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3709614977 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.487 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6807", 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3709614977 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6803/3709614977", 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6805/3709614977", 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3709614977", 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3709614977", 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T08:51:23.488 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "073cdffb-769a-4663-9f7b-8f984850bb5e", 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T08:51:23.489 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 12, 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 35, 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 0, 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6810", 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1413870850 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.490 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6811", 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1413870850 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6812", 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1413870850 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.491 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6813", 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1413870850 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.492 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6816", 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1413870850 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6817", 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1413870850 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.493 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6814", 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1413870850 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6815", 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 1413870850 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6811/1413870850", 2024-04-25T08:51:23.494 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6813/1413870850", 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1413870850", 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1413870850", 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "158029d6-c763-4e32-8bea-7175b66a64f0", 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T08:51:23.495 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 20, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 35, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 0, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.496 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6818", 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2274800911 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6819", 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2274800911 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.497 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6820", 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2274800911 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6821", 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2274800911 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T08:51:23.498 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6824", 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2274800911 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6825", 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2274800911 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.499 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6822", 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2274800911 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6823", 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 2274800911 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.500 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6819/2274800911", 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6821/2274800911", 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6825/2274800911", 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6823/2274800911", 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 3, 2024-04-25T08:51:23.501 INFO:tasks.workunit.client.0.smithi088.stdout: "uuid": "1a071d3c-4347-46da-a2e9-b20700775586", 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 1, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "in": 1, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 1, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_affinity": 1, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_begin": 0, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "last_clean_end": 0, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 33, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "up_thru": 37, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "down_at": 0, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "lost_at": 0, 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addrs": { 2024-04-25T08:51:23.502 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6826", 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3927716869 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6827", 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3927716869 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.503 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addrs": { 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6828", 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3927716869 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6829", 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3927716869 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.504 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addrs": { 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6832", 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3927716869 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6833", 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3927716869 2024-04-25T08:51:23.505 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addrs": { 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: "addrvec": [ 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v2", 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6830", 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3927716869 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "v1", 2024-04-25T08:51:23.506 INFO:tasks.workunit.client.0.smithi088.stdout: "addr": "127.0.0.1:6831", 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: "nonce": 3927716869 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: "public_addr": "127.0.0.1:6827/3927716869", 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: "cluster_addr": "127.0.0.1:6829/3927716869", 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_back_addr": "127.0.0.1:6833/3927716869", 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: "heartbeat_front_addr": "127.0.0.1:6831/3927716869", 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: "state": [ 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: "exists", 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: "up" 2024-04-25T08:51:23.507 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_xinfo": [ 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "0.000000", 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T08:46:44.742926+0000", 2024-04-25T08:51:23.508 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "0.000000", 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T08:47:18.590307+0000", 2024-04-25T08:51:23.509 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "0.000000", 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T08:47:53.632173+0000", 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.510 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 3, 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "down_stamp": "0.000000", 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_probability": 0, 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "laggy_interval": 0, 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "features": 4540701547738038271, 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "old_weight": 0, 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "last_purged_snaps_scrub": "2024-04-25T08:48:29.008181+0000", 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "dead_epoch": 0 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap": [], 2024-04-25T08:51:23.511 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_items": [ 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: "pgid": "1.1", 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: "mappings": [ 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: "from": 0, 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: "to": 1 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: "pgid": "1.4", 2024-04-25T08:51:23.512 INFO:tasks.workunit.client.0.smithi088.stdout: "mappings": [ 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: "from": 0, 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: "to": 2 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_upmap_primaries": [], 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: "pg_temp": [], 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: "primary_temp": [], 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: "blocklist": {}, 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: "range_blocklist": {}, 2024-04-25T08:51:23.513 INFO:tasks.workunit.client.0.smithi088.stdout: "erasure_code_profiles": { 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "default": { 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "k": "2", 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "m": "2", 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "plugin": "jerasure", 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "technique": "reed_sol_van" 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "removed_snaps_queue": [], 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "new_removed_snaps": [], 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "new_purged_snaps": [], 2024-04-25T08:51:23.514 INFO:tasks.workunit.client.0.smithi088.stdout: "crush_node_flags": {}, 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "device_class_flags": {}, 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode": { 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_enabled": false, 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_bucket_count": 0, 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "degraded_stretch_mode": 0, 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "recovering_stretch_mode": 0, 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "stretch_mode_bucket": 0 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_metadata": [ 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.515 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6804/3709614977,v1:127.0.0.1:6805/3709614977]", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T08:51:23.516 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/0/block", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T08:51:23.517 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T08:46:18.745367Z", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6802/3709614977,v1:127.0.0.1:6803/3709614977]", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3709614977,v1:127.0.0.1:6809/3709614977]", 2024-04-25T08:51:23.518 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3709614977,v1:127.0.0.1:6807/3709614977]", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/osd-rep-recov-eio.sh/0", 2024-04-25T08:51:23.519 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6812/1413870850,v1:127.0.0.1:6813/1413870850]", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T08:51:23.520 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/1/block", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T08:51:23.521 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T08:46:51.678798Z", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T08:51:23.522 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6810/1413870850,v1:127.0.0.1:6811/1413870850]", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1413870850,v1:127.0.0.1:6817/1413870850]", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1413870850,v1:127.0.0.1:6815/1413870850]", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T08:51:23.523 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/osd-rep-recov-eio.sh/1", 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6820/2274800911,v1:127.0.0.1:6821/2274800911]", 2024-04-25T08:51:23.524 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/2/block", 2024-04-25T08:51:23.525 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T08:51:23.526 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T08:51:23.527 INFO:tasks.workunit.client.0.smithi088.stderr:report 673814294 2024-04-25T08:51:23.527 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T08:47:26.692122Z", 2024-04-25T08:51:23.527 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T08:51:23.527 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6818/2274800911,v1:127.0.0.1:6819/2274800911]", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/2274800911,v1:127.0.0.1:6825/2274800911]", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/2274800911,v1:127.0.0.1:6823/2274800911]", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T08:51:23.528 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/osd-rep-recov-eio.sh/2", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.529 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 3, 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "arch": "x86_64", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "back_addr": "[v2:127.0.0.1:6828/3927716869,v1:127.0.0.1:6829/3927716869]", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "back_iface": "", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs": "1", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_db": "0", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "bluefs_single_shared_device": "1", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T08:51:23.530 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/3/block", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_release": "squid", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T08:51:23.531 INFO:tasks.workunit.client.0.smithi088.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "created_at": "2024-04-25T08:48:00.780338Z", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "default_device_class": "hdd", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHZ6", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": "sda", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "distro": "ubuntu", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "distro_version": "22.04", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "front_addr": "[v2:127.0.0.1:6826/3927716869,v1:127.0.0.1:6827/3927716869]", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "front_iface": "", 2024-04-25T08:51:23.532 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_back_addr": "[v2:127.0.0.1:6832/3927716869,v1:127.0.0.1:6833/3927716869]", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_front_addr": "[v2:127.0.0.1:6830/3927716869,v1:127.0.0.1:6831/3927716869]", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "hostname": "smithi088", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "journal_rotational": "1", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_swap_kb": "2097148", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "mem_total_kb": "32746872", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "os": "Linux", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_data": "td/osd-rep-recov-eio.sh/3", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_objectstore": "bluestore", 2024-04-25T08:51:23.533 INFO:tasks.workunit.client.0.smithi088.stdout: "osdspec_affinity": "", 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: "rotational": "1" 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_clean_epochs": { 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: "min_last_epoch_clean": 0, 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: "last_epoch_clean": { 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: "per_pool": [ 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: "floor": 4294967295 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.534 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_epochs": [ 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 8 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 13 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 20 2024-04-25T08:51:23.535 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 3, 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 33 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_first_committed": 1, 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: "osdmap_last_committed": 39, 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: "crushmap": { 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: "devices": [ 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T08:51:23.536 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.0", 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.1", 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.2", 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.537 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 3, 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd.3", 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: "class": "hdd" 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: "types": [ 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 0, 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "osd" 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.538 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "host" 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 2, 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "chassis" 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 3, 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "rack" 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 4, 2024-04-25T08:51:23.539 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "row" 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 5, 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pdu" 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 6, 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "pod" 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 7, 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "room" 2024-04-25T08:51:23.540 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 8, 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "datacenter" 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 9, 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "zone" 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 10, 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "region" 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.541 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "root" 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "buckets": [ 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -1, 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default", 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 25608, 2024-04-25T08:51:23.542 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 25608, 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -2, 2024-04-25T08:51:23.543 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "default~hdd", 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 11, 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "root", 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 25608, 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 25608, 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.544 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -3, 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088", 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 25608, 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T08:51:23.545 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.546 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 3, 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 3 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "id": -4, 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "name": "smithi088~hdd", 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "type_id": 1, 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "type_name": "host", 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 25608, 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "alg": "straw2", 2024-04-25T08:51:23.547 INFO:tasks.workunit.client.0.smithi088.stdout: "hash": "rjenkins1", 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: "items": [ 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 0, 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 0 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 1, 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 1 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.548 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 2, 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 2 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: "id": 3, 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: "weight": 6402, 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: "pos": 3 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: "rules": [ 2024-04-25T08:51:23.549 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_id": 0, 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: "rule_name": "replicated_rule", 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: "type": 1, 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: "steps": [ 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "take", 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: "item": -1, 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: "item_name": "default" 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.550 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "choose_firstn", 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 0, 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: "type": "osd" 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: "op": "emit" 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: ] 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: "tunables": { 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_tries": 0, 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_local_fallback_tries": 0, 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_total_tries": 50, 2024-04-25T08:51:23.551 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_descend_once": 1, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_vary_r": 1, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "chooseleaf_stable": 1, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_descents": 100, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "msr_collision_tries": 100, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "straw_calc_version": 1, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "allowed_bucket_algs": 54, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "profile": "jewel", 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "optimal_tunables": 1, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "legacy_tunables": 0, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "minimum_required_version": "jewel", 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables": 1, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables2": 1, 2024-04-25T08:51:23.552 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v2_rules": 0, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables3": 1, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v3_rules": 0, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v4_buckets": 1, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "require_feature_tunables5": 1, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "has_v5_rules": 0, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "has_msr_rules": 0 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "choose_args": {} 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "fsmap": { 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 1, 2024-04-25T08:51:23.553 INFO:tasks.workunit.client.0.smithi088.stdout: "default_fscid": -1, 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": { 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "compat": {}, 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "ro_compat": {}, 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "incompat": { 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_1": "base v0.20", 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_2": "client writeable ranges", 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_4": "dir inode in separate object", 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T08:51:23.554 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_8": "no anchor table", 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_9": "file layout v2", 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_10": "snaprealm v2", 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_11": "minor log segments", 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "feature_flags": { 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "enable_multiple": true, 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "ever_enabled_multiple": true 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "standbys": [], 2024-04-25T08:51:23.555 INFO:tasks.workunit.client.0.smithi088.stdout: "filesystems": [] 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_first_committed": 1, 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "mdsmap_last_committed": 1, 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "auth": { 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 1, 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 6, 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_secrets": 4 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 9, 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_active": 9, 2024-04-25T08:51:23.556 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_unknown": 0, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osd": 4, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_sum": { 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 4096, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 1, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 3, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 1, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 3, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 1, 2024-04-25T08:51:23.557 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 1, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 3, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 12, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 1, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 4, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 3, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 12288, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T08:51:23.558 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T08:51:23.559 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 3 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T08:51:23.560 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 1, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 1, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 27, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 27, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 0 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.561 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum": { 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 27, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 4, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 4, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 4, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 419430400, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 108604, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 1292, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 6, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 107257, 2024-04-25T08:51:23.562 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 419321796, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 429496729600, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 429385519104, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1323008, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 450436, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 6378, 2024-04-25T08:51:23.563 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 109831958 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 4, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [ 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.564 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.565 INFO:tasks.workunit.client.0.smithi088.stdout: 1 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 262144 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T08:51:23.566 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_sum_by_class": { 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "hdd": { 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "up_from": 0, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 0, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pgs": 27, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "num_osds": 4, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_osds": 4, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "num_per_pool_omap_osds": 4, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "kb": 419430400, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used": 108604, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_data": 1292, 2024-04-25T08:51:23.567 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_omap": 6, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_used_meta": 107257, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "kb_avail": 419321796, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "statfs": { 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 429496729600, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 429385519104, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 1323008, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 450436, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 6378, 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 109831958 2024-04-25T08:51:23.568 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: "hb_peers": [], 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: "snap_trim_queue_len": 0, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: "num_snap_trimming": 0, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shards_repaired": 4, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: "op_queue_age_hist": { 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: "histogram": [ 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.569 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.570 INFO:tasks.workunit.client.0.smithi088.stdout: 0, 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: 1 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: "upper_bound": 262144 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: "perf_stat": { 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ms": 0, 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ms": 0, 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: "commit_latency_ns": 0, 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: "apply_latency_ns": 0 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: "alerts": [], 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: "network_ping_times": [] 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.571 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "pool_stats": [ 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 2, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 1, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 4096, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 1, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 3, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 1, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 3, 2024-04-25T08:51:23.572 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 1, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 1, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 3, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 12, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 1, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 4, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 3, 2024-04-25T08:51:23.573 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 12288, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T08:51:23.574 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 3 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T08:51:23.575 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 12288, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 12288, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 1, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 1, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 3, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 3, 2024-04-25T08:51:23.576 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 3 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "poolid": 1, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg": 8, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "stat_sum": { 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes": 0, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects": 0, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_clones": 0, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_object_copies": 0, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_missing": 0, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_degraded": 0, 2024-04-25T08:51:23.577 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_misplaced": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_unfound": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_dirty": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_whiteouts": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_read_kb": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_write_kb": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_scrub_errors": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_deep_scrub_errors": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_recovered": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_recovered": 0, 2024-04-25T08:51:23.578 INFO:tasks.workunit.client.0.smithi088.stdout: "num_keys_recovered": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_omap": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_kb": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_kb": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_promote": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_high": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_flush_mode_low": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_some": 0, 2024-04-25T08:51:23.579 INFO:tasks.workunit.client.0.smithi088.stdout: "num_evict_mode_full": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_pinned": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "num_legacy_snapsets": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "num_large_omap_objects": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_manifest": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_bytes": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "num_omap_keys": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "num_objects_repaired": 0 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "store_stats": { 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "total": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "available": 0, 2024-04-25T08:51:23.580 INFO:tasks.workunit.client.0.smithi088.stdout: "internally_reserved": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "allocated": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "data_stored": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_allocated": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "data_compressed_original": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "omap_allocated": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "internal_metadata": 0 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "log_size": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "ondisk_log_size": 0, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "up": 24, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "acting": 24, 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: "num_store_stats": 4 2024-04-25T08:51:23.581 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: "osd_stats": [ 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 25769803853 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 51539607621 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.582 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 85899345981 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 3, 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: "seq": 141733920824 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_state": [ 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+recovery_unfound+degraded", 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 1 2024-04-25T08:51:23.583 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "state": "active+clean", 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "num": 8 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "num_pg_by_osd": [ 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 3, 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 4, 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 7, 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 2, 2024-04-25T08:51:23.584 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 1, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 6, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 1, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 3, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 7, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "osd": 0, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "num_primary_pg": 1, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "num_acting_pg": 7, 2024-04-25T08:51:23.585 INFO:tasks.workunit.client.0.smithi088.stdout: "num_up_not_acting_pg": 0 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": [ 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 1, 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: { 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: "pool": 2, 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: "purged_snaps": {} 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.586 INFO:tasks.workunit.client.0.smithi088.stdout: ], 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "servicemap": { 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "epoch": 6, 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "modified": "2024-04-25T08:49:31.507949+0000", 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "services": {} 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: }, 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_first_committed": 1, 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "mgrstat_last_committed": 144, 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_first_committed": 1, 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "logm_last_committed": 650, 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "paxos": { 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "first_committed": 252, 2024-04-25T08:51:23.587 INFO:tasks.workunit.client.0.smithi088.stdout: "last_committed": 837, 2024-04-25T08:51:23.588 INFO:tasks.workunit.client.0.smithi088.stdout: "last_pn": 0, 2024-04-25T08:51:23.588 INFO:tasks.workunit.client.0.smithi088.stdout: "accepted_pn": 0 2024-04-25T08:51:23.588 INFO:tasks.workunit.client.0.smithi088.stdout: } 2024-04-25T08:51:23.588 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T08:51:23.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T08:51:23.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: ceph pg 2.0 query 2024-04-25T08:51:23.589 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T08:51:23.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: COUNT=3 2024-04-25T08:51:23.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:161: rados_get_data: test 3 = 3 2024-04-25T08:51:23.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:162: rados_get_data: flush_pg_stats 2024-04-25T08:51:23.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:51:23.629 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:51:24.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:51:24.041 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:51:24.041 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:51:24.041 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:51:24.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:51:24.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:24.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:51:24.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803855 2024-04-25T08:51:24.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803855 2024-04-25T08:51:24.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803855' 2024-04-25T08:51:24.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:24.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:51:24.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607623 2024-04-25T08:51:24.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607623 2024-04-25T08:51:24.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803855 1-51539607623' 2024-04-25T08:51:24.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:24.370 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:51:24.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345983 2024-04-25T08:51:24.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345983 2024-04-25T08:51:24.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803855 1-51539607623 2-85899345983' 2024-04-25T08:51:24.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:24.514 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:51:24.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920826 2024-04-25T08:51:24.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920826 2024-04-25T08:51:24.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803855 1-51539607623 2-85899345983 3-141733920826' 2024-04-25T08:51:24.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:24.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803855 2024-04-25T08:51:24.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:24.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:51:24.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803855 2024-04-25T08:51:24.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:24.654 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803855 2024-04-25T08:51:24.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803855 2024-04-25T08:51:24.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803855' 2024-04-25T08:51:24.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:51:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803853 -lt 25769803855 2024-04-25T08:51:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:51:26.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:51:26.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:51:26.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803856 -lt 25769803855 2024-04-25T08:51:26.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:26.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607623 2024-04-25T08:51:26.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:26.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:51:26.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607623 2024-04-25T08:51:26.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:26.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607623 2024-04-25T08:51:26.467 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607623 2024-04-25T08:51:26.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607623' 2024-04-25T08:51:26.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:51:26.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607624 -lt 51539607623 2024-04-25T08:51:26.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:26.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345983 2024-04-25T08:51:26.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:26.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:51:26.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345983 2024-04-25T08:51:26.891 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:26.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345983 2024-04-25T08:51:26.892 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345983 2024-04-25T08:51:26.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345983' 2024-04-25T08:51:26.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:51:27.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345984 -lt 85899345983 2024-04-25T08:51:27.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:27.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:27.318 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-141733920826 2024-04-25T08:51:27.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:51:27.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-141733920826 2024-04-25T08:51:27.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:27.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920826 2024-04-25T08:51:27.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 141733920826' 2024-04-25T08:51:27.322 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 141733920826 2024-04-25T08:51:27.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:51:27.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920827 -lt 141733920826 2024-04-25T08:51:27.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T08:51:27.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T08:51:28.135 INFO:tasks.workunit.client.0.smithi088.stderr:dumped all 2024-04-25T08:51:28.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: COUNT=4 2024-04-25T08:51:28.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:164: rados_get_data: test 4 = 4 2024-04-25T08:51:28.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:177: TEST_rados_get_with_eio: delete_pool pool-rep 2024-04-25T08:51:28.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-04-25T08:51:28.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-04-25T08:51:28.706 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'pool-rep' does not exist 2024-04-25T08:51:28.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T08:51:28.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:51:28.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:51:28.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T08:51:28.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:51:28.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:51:28.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:51:28.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:51:28.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:51:29.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:51:29.036 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:51:29.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:51:29.037 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:51:29.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:51:29.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:51:29.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:51:29.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:51:29.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:51:29.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:51:29.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:51:29.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:51:29.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:51:29.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T08:51:29.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:51:29.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:29.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:51:29.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:51:29.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:51:29.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:51:29.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:51:29.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:51:29.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:51:29.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:51:29.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:51:29.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:51:29.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:51:29.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:51:29.085 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:51:29.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:51:29.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:51:29.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:51:29.086 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:51:29.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:51:29.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:51:29.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T08:51:29.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:51:29.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:29.090 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:51:29.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T08:51:29.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:51:29.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:51:29.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T08:51:29.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:51:29.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:29.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:51:29.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.389694 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:51:29.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T08:51:29.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T08:51:29.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:51:29.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:51:29.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:51:29.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:51:29.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:29.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:51:29.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:51:29.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T08:51:29.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:51:29.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:51:29.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:51:29.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:51:29.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:51:29.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:51:29.804 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:51:29.804 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:51:29.804 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:51:29.804 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:51:29.804 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:29.805 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:51:29.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-mon.a.asok 2024-04-25T08:51:29.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:51:29.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.389694/ceph-mon.a.asok config get fsid 2024-04-25T08:51:29.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:51:29.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:51:29.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:51:29.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:51:29.919 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:51:29.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:51:29.919 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:51:29.919 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:51:29.919 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:51:29.919 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:29.920 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:51:29.920 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-mon.a.asok 2024-04-25T08:51:29.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:51:29.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.389694/ceph-mon.a.asok config get mon_host 2024-04-25T08:51:30.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T08:51:30.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:51:30.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:51:30.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:51:30.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:51:30.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T08:51:30.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:51:30.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:51:30.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:51:30.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:51:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:51:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:30.963 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:51:30.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:51:30.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:51:30.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:51:30.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T08:51:31.300 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T08:51:31.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_repair_warning td/osd-rep-recov-eio.sh 2024-04-25T08:51:31.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:181: TEST_rados_repair_warning: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:51:31.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: expr 10 + 1 2024-04-25T08:51:31.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: local OBJS=11 2024-04-25T08:51:31.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:184: TEST_rados_repair_warning: setup_osds 4 2024-04-25T08:51:31.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-04-25T08:51:31.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T08:51:31.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T08:51:31.319 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-04-25T08:51:31.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-04-25T08:51:31.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:51:31.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T08:51:31.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:51:31.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:51:31.324 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:51:31.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:51:31.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:51:31.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:51:31.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:51:31.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T08:51:31.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:51:31.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=21866187-4132-4111-ad47-99bec405f41e 2024-04-25T08:51:31.327 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 21866187-4132-4111-ad47-99bec405f41e 2024-04-25T08:51:31.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 21866187-4132-4111-ad47-99bec405f41e' 2024-04-25T08:51:31.327 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:51:31.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTGSpmTEr+FBAAHitgK5uYJuJUbG/4SW0O+A== 2024-04-25T08:51:31.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTGSpmTEr+FBAAHitgK5uYJuJUbG/4SW0O+A=="}' 2024-04-25T08:51:31.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 21866187-4132-4111-ad47-99bec405f41e -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:51:31.676 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:51:31.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:51:31.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTGSpmTEr+FBAAHitgK5uYJuJUbG/4SW0O+A== --osd-uuid 21866187-4132-4111-ad47-99bec405f41e 2024-04-25T08:51:31.719 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:31.726+0000 7f4523d2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:31.753 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:31.758+0000 7f4523d2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:31.754 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:31.762+0000 7f4523d2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:31.754 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:31.762+0000 7f4523d2f7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:51:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:31.762+0000 7f4523d2f7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:51:31.755 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:31.762+0000 7f4523d2f7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:51:31.792 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:31.798+0000 7f4523d2f7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T08:51:37.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T08:51:37.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:51:37.901 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:51:37.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:51:37.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:51:38.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:51:38.455 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:51:38.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:51:38.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:51:38.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:51:38.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:51:38.483 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:38.490+0000 7f24216f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:38.486 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:38.490+0000 7f24216f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:38.487 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:38.494+0000 7f24216f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:38.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:51:38.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:51:38.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:51:38.854 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:51:38.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:51:38.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:51:38.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:38.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:51:38.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:38.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:39.031 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:39.038+0000 7f24216f67c0 -1 Falling back to public interface 2024-04-25T08:51:39.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:40.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:40.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:40.319 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:51:40.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:51:40.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:40.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:40.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:41.676 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:51:41.682+0000 7f24216f67c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:51:41.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:41.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:41.739 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:51:41.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:51:41.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:41.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:42.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:43.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:43.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:43.383 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:51:43.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:51:43.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:43.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:43.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:44.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:44.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:44.851 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:51:44.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:51:44.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:44.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:45.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:46.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:46.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:46.304 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:51:46.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:51:46.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:46.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:46.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:47.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:47.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:47.837 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:51:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:51:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:47.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:48.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:51:49.283 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:51:49.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:49.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:49.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:50.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:50.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:50.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:51:50.779 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:51:50.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:50.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:51.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:52.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:52.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:52.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:51:52.231 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:51:52.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:52.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:52.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:53.707 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:51:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:51:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:54.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:55.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:55.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:55.159 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:51:55.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:51:55.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:55.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:56.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:57.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:57.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:57.103 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:51:57.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:51:57.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:57.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:57.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:58.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:58.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:58.953 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:51:58.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:51:58.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:58.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:59.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:00.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:00.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:00.831 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:52:00.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:52:00.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:00.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:52:01.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:02.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:02.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:02.705 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:52:02.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:52:02.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:02.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:52:03.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:04.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:04.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:04.502 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:52:04.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:52:04.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:04.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:52:05.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:06.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:06.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:06.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:52:06.330 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:52:06.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:06.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:52:06.786 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3352894471,v1:127.0.0.1:6803/3352894471] [v2:127.0.0.1:6804/3352894471,v1:127.0.0.1:6805/3352894471] exists,up 21866187-4132-4111-ad47-99bec405f41e 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:52:06.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:52:06.793 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:52:06.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:52:06.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:52:06.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:52:06.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T08:52:06.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:52:06.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae28e2ea-5ba7-4fac-85ee-1555392ca23a 2024-04-25T08:52:06.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ae28e2ea-5ba7-4fac-85ee-1555392ca23a' 2024-04-25T08:52:06.796 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 ae28e2ea-5ba7-4fac-85ee-1555392ca23a 2024-04-25T08:52:06.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:52:06.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2GSpm2/9BMRAA5m+PJpA29H+1tTF25JGnRg== 2024-04-25T08:52:06.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2GSpm2/9BMRAA5m+PJpA29H+1tTF25JGnRg=="}' 2024-04-25T08:52:06.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae28e2ea-5ba7-4fac-85ee-1555392ca23a -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:52:07.654 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:52:07.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:52:07.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2GSpm2/9BMRAA5m+PJpA29H+1tTF25JGnRg== --osd-uuid ae28e2ea-5ba7-4fac-85ee-1555392ca23a 2024-04-25T08:52:07.707 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:07.717+0000 7f22473117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:07.725 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:07.733+0000 7f22473117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:07.726 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:07.737+0000 7f22473117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:07.727 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:07.737+0000 7f22473117c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:52:07.727 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:07.737+0000 7f22473117c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:52:07.727 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:07.737+0000 7f22473117c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:52:07.817 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:07.825+0000 7f22473117c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T08:52:14.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T08:52:14.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:52:14.280 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:52:14.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:52:14.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:52:14.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:52:14.824 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:52:14.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:52:14.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:52:14.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:52:14.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:52:14.853 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:14.861+0000 7f244eecf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:14.855 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:14.865+0000 7f244eecf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:14.856 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:14.865+0000 7f244eecf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:52:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:52:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:52:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:52:15.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:52:15.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:15.242 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:52:15.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:52:15.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:15.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:15.423 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:15.433+0000 7f244eecf7c0 -1 Falling back to public interface 2024-04-25T08:52:15.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:16.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:16.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:16.670 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:52:16.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:52:16.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:16.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:17.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:18.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:18.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:18.099 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:52:18.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:52:18.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:18.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:18.233 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:18.241+0000 7f244eecf7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:52:18.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:19.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:19.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:19.607 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:52:19.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:52:19.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:19.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:20.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:21.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:21.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:21.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:52:21.211 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:52:21.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:21.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:21.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:22.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:22.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:22.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:52:22.671 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:52:22.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:22.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:23.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:52:24.157 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:52:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:24.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:25.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:25.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:25.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:52:25.587 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:52:25.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:25.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:26.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:27.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:27.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:27.184 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:52:27.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:52:27.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:27.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:27.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:28.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:28.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:28.639 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:52:28.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:52:28.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:28.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:29.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:30.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:30.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:30.149 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:52:30.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:52:30.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:30.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:30.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:31.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:31.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:31.669 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:52:31.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:52:31.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:31.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:32.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:33.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:33.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:33.270 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:52:33.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:52:33.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:33.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:34.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:35.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:35.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:35.497 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:52:35.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:52:35.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:35.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:36.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:37.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:37.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:37.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:52:37.280 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:52:37.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:37.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:38.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:39.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:39.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:39.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:52:39.057 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:52:39.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:39.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:40.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:41.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:41.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:41.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:52:41.238 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:52:41.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:41.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:41.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:43.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:43.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:43.000 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:52:43.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:52:43.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:43.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:43.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:44.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:44.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:44.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:52:44.427 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:52:44.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:44.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:44.920 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/592021960,v1:127.0.0.1:6811/592021960] [v2:127.0.0.1:6812/592021960,v1:127.0.0.1:6813/592021960] exists,up ae28e2ea-5ba7-4fac-85ee-1555392ca23a 2024-04-25T08:52:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:52:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:52:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:52:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:52:44.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:52:44.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:52:44.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ad600bc1-ee84-43cd-be53-4dc62c3827ea 2024-04-25T08:52:44.931 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 ad600bc1-ee84-43cd-be53-4dc62c3827ea 2024-04-25T08:52:44.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ad600bc1-ee84-43cd-be53-4dc62c3827ea' 2024-04-25T08:52:44.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:52:44.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcGSpmBQYOORAAN+JACwtIOOffl3XZExrZ4g== 2024-04-25T08:52:44.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcGSpmBQYOORAAN+JACwtIOOffl3XZExrZ4g=="}' 2024-04-25T08:52:44.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ad600bc1-ee84-43cd-be53-4dc62c3827ea -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T08:52:45.420 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:52:45.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T08:52:45.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcGSpmBQYOORAAN+JACwtIOOffl3XZExrZ4g== --osd-uuid ad600bc1-ee84-43cd-be53-4dc62c3827ea 2024-04-25T08:52:45.469 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:45.477+0000 7f612f0f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:45.495 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:45.505+0000 7f612f0f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:45.496 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:45.505+0000 7f612f0f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:45.497 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:45.505+0000 7f612f0f87c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:52:45.497 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:45.505+0000 7f612f0f87c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:52:45.497 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:45.505+0000 7f612f0f87c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:52:45.535 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:45.545+0000 7f612f0f87c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T08:52:52.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T08:52:52.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:52:52.284 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:52:52.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:52:52.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:52:52.844 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:52:52.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:52:52.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:52:52.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:52:52.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:52:52.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:52:52.873 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:52.880+0000 7f7d26c5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:52.876 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:52.884+0000 7f7d26c5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:52.877 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:52.884+0000 7f7d26c5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:52:53.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:52:53.279 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:52:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:52:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:52:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:52:53.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:53.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:52:53.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:53.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:53.455 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:53.464+0000 7f7d26c5d7c0 -1 Falling back to public interface 2024-04-25T08:52:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:54.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:54.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:54.707 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:52:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:52:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:54.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:55.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:56.068 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:52:56.076+0000 7f7d26c5d7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:52:56.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:56.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:56.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:52:56.133 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:52:56.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:56.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:56.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:57.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:57.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:57.734 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:52:57.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:52:57.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:57.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:58.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:59.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:59.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:59.460 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:52:59.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:52:59.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:59.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:59.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:00.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:00.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:00.860 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:53:00.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:53:00.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:00.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:01.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:02.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:02.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:02.385 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:53:02.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:53:02.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:02.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:02.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:03.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:03.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:03.852 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:53:03.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:53:03.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:03.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:04.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:05.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:05.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:05.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:53:05.426 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:53:05.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:05.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:05.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:06.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:06.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:06.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:53:06.885 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:53:06.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:06.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:07.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:08.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:08.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:08.399 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:53:08.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:53:08.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:08.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:08.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:09.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:09.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:09.891 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:53:09.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:53:09.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:09.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:10.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:11.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:11.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:11.760 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:53:11.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:53:11.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:11.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:12.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:13.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:13.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:13.270 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:53:13.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:53:13.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:13.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:14.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:15.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:15.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:15.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:53:15.169 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:53:15.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:15.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:15.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:16.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:16.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:16.987 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:53:16.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:53:16.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:16.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:17.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:18.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:18.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:18.784 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:53:18.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:53:18.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:18.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:19.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:20.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:20.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:20.687 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:53:20.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:53:20.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:20.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:21.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:22.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:22.284 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:53:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:53:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:22.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:22.878 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 21 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3511680388,v1:127.0.0.1:6819/3511680388] [v2:127.0.0.1:6820/3511680388,v1:127.0.0.1:6821/3511680388] exists,up ad600bc1-ee84-43cd-be53-4dc62c3827ea 2024-04-25T08:53:22.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:53:22.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:53:22.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:53:22.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:53:22.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-04-25T08:53:22.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:53:22.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:53:22.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:53:22.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:53:22.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-04-25T08:53:22.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T08:53:22.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:53:22.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:53:22.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:53:22.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-04-25T08:53:22.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-04-25T08:53:22.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:53:22.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:53:22.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:53:22.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:53:22.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:22.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:22.887 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:22.887 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:22.887 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:53:22.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:53:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:53:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:53:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:53:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:53:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-04-25T08:53:22.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:53:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bfb8fc7f-1206-40b8-a7f8-e338662791f1 2024-04-25T08:53:22.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bfb8fc7f-1206-40b8-a7f8-e338662791f1' 2024-04-25T08:53:22.890 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 bfb8fc7f-1206-40b8-a7f8-e338662791f1 2024-04-25T08:53:22.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:53:22.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACGipm3tuENhAAabuKoCdMsuItYjpoqsSSNw== 2024-04-25T08:53:22.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACGipm3tuENhAAabuKoCdMsuItYjpoqsSSNw=="}' 2024-04-25T08:53:22.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bfb8fc7f-1206-40b8-a7f8-e338662791f1 -i td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T08:53:23.623 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:53:23.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T08:53:23.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACGipm3tuENhAAabuKoCdMsuItYjpoqsSSNw== --osd-uuid bfb8fc7f-1206-40b8-a7f8-e338662791f1 2024-04-25T08:53:23.668 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:23.676+0000 7f6a1760c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:23.670 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:23.680+0000 7f6a1760c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:23.671 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:23.680+0000 7f6a1760c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:23.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:23.680+0000 7f6a1760c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:53:23.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:23.680+0000 7f6a1760c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:53:23.672 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:23.680+0000 7f6a1760c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T08:53:23.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:23.824+0000 7f6a1760c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-04-25T08:53:30.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-04-25T08:53:30.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:53:30.332 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T08:53:30.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:53:30.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:53:30.916 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T08:53:30.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:53:30.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:53:30.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:53:30.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:53:30.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:53:30.946 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:30.956+0000 7f2e4c21e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:30.955 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:30.964+0000 7f2e4c21e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:30.956 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:30.964+0000 7f2e4c21e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:31.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:53:31.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:53:31.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:53:31.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:53:31.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:53:31.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:31.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:53:31.340 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:53:31.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:31.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:31.511 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:31.520+0000 7f2e4c21e7c0 -1 Falling back to public interface 2024-04-25T08:53:31.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:32.789 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:53:32.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:53:32.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:32.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:33.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:34.159 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:53:34.168+0000 7f2e4c21e7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:53:34.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:34.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:34.209 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:53:34.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:53:34.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:34.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:34.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:35.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:35.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:35.636 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:53:35.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:53:35.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:35.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:36.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:37.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:37.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:37.289 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:53:37.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:53:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:37.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:37.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:38.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:38.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:38.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:53:38.866 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:53:38.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:38.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:39.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:40.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:40.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:40.556 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:53:40.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:53:40.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:40.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:40.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:41.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:41.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:41.988 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:53:41.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:53:41.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:41.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:42.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:43.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:43.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:43.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:53:43.625 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:53:43.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:43.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:44.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:45.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:45.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:45.173 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:53:45.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:53:45.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:45.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:45.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:46.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:46.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:46.597 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:53:46.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:53:46.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:46.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:47.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:48.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:48.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:48.169 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:53:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:53:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:48.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:49.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:49.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:49.749 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:53:49.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:53:49.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:49.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:50.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:51.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:51.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:51.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:53:51.739 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:53:51.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:51.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:52.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:53.701 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:53:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:53:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:54.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:55.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:55.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:55.559 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:53:55.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:53:55.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:55.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:56.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:57.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:57.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:57.690 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:53:57.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:53:57.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:57.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:58.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:59.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:59.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:59.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:53:59.897 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:53:59.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:59.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:54:00.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:01.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:54:01.316 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:54:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:54:01.892 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2499423740,v1:127.0.0.1:6827/2499423740] [v2:127.0.0.1:6828/2499423740,v1:127.0.0.1:6829/2499423740] exists,up bfb8fc7f-1206-40b8-a7f8-e338662791f1 2024-04-25T08:54:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:54:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:54:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:54:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T08:54:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:54:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:54:01.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:54:01.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:54:01.895 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:54:01.895 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:54:01.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:54:01.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:54:01.895 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:54:02.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:54:02.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:54:02.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:54:02.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:54:02.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:54:02.006 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:54:02.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:54:02.703 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:02.703 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:54:02.703 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:54:02.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:54:02.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:02.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:54:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803804 2024-04-25T08:54:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803804 2024-04-25T08:54:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804' 2024-04-25T08:54:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:02.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:54:03.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574872 2024-04-25T08:54:03.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574872 2024-04-25T08:54:03.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-55834574872' 2024-04-25T08:54:03.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:03.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:54:03.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313234 2024-04-25T08:54:03.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313234 2024-04-25T08:54:03.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-55834574872 2-90194313234' 2024-04-25T08:54:03.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:03.371 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:54:03.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953483 2024-04-25T08:54:03.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953483 2024-04-25T08:54:03.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-55834574872 2-90194313234 3-137438953483' 2024-04-25T08:54:03.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:03.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803804 2024-04-25T08:54:03.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:03.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:54:03.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803804 2024-04-25T08:54:03.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:03.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803804 2024-04-25T08:54:03.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2024-04-25T08:54:03.617 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803804 2024-04-25T08:54:03.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:04.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803804 -lt 25769803804 2024-04-25T08:54:04.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:04.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:04.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574872 2024-04-25T08:54:04.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:54:04.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574872 2024-04-25T08:54:04.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:04.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574872 2024-04-25T08:54:04.044 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574872 2024-04-25T08:54:04.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574872' 2024-04-25T08:54:04.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:54:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574872 -lt 55834574872 2024-04-25T08:54:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:04.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313234 2024-04-25T08:54:04.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:04.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:54:04.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313234 2024-04-25T08:54:04.507 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:04.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313234 2024-04-25T08:54:04.509 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 90194313234 2024-04-25T08:54:04.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313234' 2024-04-25T08:54:04.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:54:04.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313234 -lt 90194313234 2024-04-25T08:54:04.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:04.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953483 2024-04-25T08:54:04.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:04.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:54:04.942 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953483 2024-04-25T08:54:04.942 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:04.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953483 2024-04-25T08:54:04.943 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 137438953483 2024-04-25T08:54:04.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953483' 2024-04-25T08:54:04.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:54:05.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953483 -lt 137438953483 2024-04-25T08:54:05.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:54:05.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:05.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:05.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T08:54:05.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:54:05.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:54:05.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:54:05.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:54:05.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:54:05.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:54:05.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:54:06.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-25T08:54:06.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:54:06.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:06.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:06.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 8 2024-04-25T08:54:06.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 6 '!=' -1 2024-04-25T08:54:06.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:54:06.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=6 2024-04-25T08:54:06.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:54:06.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:54:07.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:54:07.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:54:07.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:54:07.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:54:07.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:54:07.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:54:07.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:54:07.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:54:07.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-25T08:54:07.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:54:07.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:07.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 8 2024-04-25T08:54:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 6 '!=' 6 2024-04-25T08:54:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:54:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:54:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:54:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:54:07.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:54:07.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:54:07.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:54:08.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:54:08.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:54:08.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T08:54:08.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:54:08.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T08:54:08.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:54:08.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:54:08.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:54:08.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:54:08.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:54:08.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:54:08.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:54:08.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:54:09.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-04-25T08:54:09.105 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:54:09.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:09.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-04-25T08:54:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 6 2024-04-25T08:54:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:54:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=7 2024-04-25T08:54:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:54:09.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:54:09.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:54:09.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:54:09.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:54:09.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:54:09.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:54:09.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:54:09.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:54:09.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:54:10.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:54:10.127 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:54:10.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:10.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:10.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T08:54:10.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:54:10.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:54:10.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:186: TEST_rados_repair_warning: local poolname=pool-rep 2024-04-25T08:54:10.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:187: TEST_rados_repair_warning: create_pool pool-rep 1 1 2024-04-25T08:54:10.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-04-25T08:54:11.209 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'pool-rep' already exists 2024-04-25T08:54:11.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:54:12.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:188: TEST_rados_repair_warning: wait_for_clean 2024-04-25T08:54:12.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:54:12.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:54:12.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:54:12.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:54:12.229 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:54:12.229 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:54:12.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:54:12.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:54:12.230 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:54:12.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:54:12.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:54:12.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:54:12.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:54:12.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:54:12.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:54:12.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:54:12.935 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:12.935 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:54:12.935 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:54:12.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:54:12.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:12.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:54:13.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803808 2024-04-25T08:54:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803808 2024-04-25T08:54:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803808' 2024-04-25T08:54:13.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:13.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:54:13.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574876 2024-04-25T08:54:13.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574876 2024-04-25T08:54:13.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803808 1-55834574876' 2024-04-25T08:54:13.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:13.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:54:13.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313238 2024-04-25T08:54:13.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313238 2024-04-25T08:54:13.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803808 1-55834574876 2-90194313238' 2024-04-25T08:54:13.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:13.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:54:13.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953487 2024-04-25T08:54:13.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953487 2024-04-25T08:54:13.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803808 1-55834574876 2-90194313238 3-137438953487' 2024-04-25T08:54:13.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:13.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803808 2024-04-25T08:54:13.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:13.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:54:13.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803808 2024-04-25T08:54:13.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:13.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803808 2024-04-25T08:54:13.539 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803808 2024-04-25T08:54:13.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-04-25T08:54:13.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:13.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803808 -lt 25769803808 2024-04-25T08:54:13.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:13.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574876 2024-04-25T08:54:13.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:13.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:54:13.974 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574876 2024-04-25T08:54:13.975 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:13.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574876 2024-04-25T08:54:13.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2024-04-25T08:54:13.976 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574876 2024-04-25T08:54:13.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:54:14.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574876 -lt 55834574876 2024-04-25T08:54:14.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:14.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313238 2024-04-25T08:54:14.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:14.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:54:14.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313238 2024-04-25T08:54:14.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:14.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313238 2024-04-25T08:54:14.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313238' 2024-04-25T08:54:14.460 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 90194313238 2024-04-25T08:54:14.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:54:14.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313238 -lt 90194313238 2024-04-25T08:54:14.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:14.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:14.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953487 2024-04-25T08:54:14.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:54:14.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953487 2024-04-25T08:54:14.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:14.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953487 2024-04-25T08:54:14.910 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 137438953487 2024-04-25T08:54:14.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953487' 2024-04-25T08:54:14.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:54:15.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953487 -lt 137438953487 2024-04-25T08:54:15.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:54:15.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:15.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:15.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:54:15.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:54:15.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:54:15.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:54:15.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:54:15.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:54:15.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:54:15.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:54:16.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:54:16.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:54:16.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:16.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:16.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:54:16.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:54:16.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:54:16.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:190: TEST_rados_repair_warning: local poolname=pool-rep 2024-04-25T08:54:16.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:191: TEST_rados_repair_warning: local objbase=obj-warn 2024-04-25T08:54:16.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:192: TEST_rados_repair_warning: local inject=eio 2024-04-25T08:54:16.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: seq 1 11 2024-04-25T08:54:16.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:16.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T08:54:16.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:16.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-1 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:16.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:17.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:17.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:17.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:17.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T08:54:17.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:17.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T08:54:17.076 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:17.076 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T08:54:17.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:17.496 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:17.496 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:17.496 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:17.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:17.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:17.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:17.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:17.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:17.497 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:17.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:17.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:17.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:17.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:17.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:17.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:17.499 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:17.499 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:17.499 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:17.499 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:17.499 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:17.499 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:17.500 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:17.500 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:17.500 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:17.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:17.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:17.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:17.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:17.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:17.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:17.623 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:17.623 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:17.623 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:17.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:17.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:17.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-04-25T08:54:17.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T08:54:17.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:17.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:17.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-04-25T08:54:17.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:17.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:17.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:17.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:17.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:17.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:17.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:17.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T08:54:17.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:17.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:17.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-2 2024-04-25T08:54:17.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:17.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:17.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:17.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:17.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:17.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:17.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:17.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:17.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:17.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:17.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T08:54:17.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:17.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T08:54:17.971 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T08:54:17.971 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:18.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:18.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:18.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:18.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:18.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:18.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:18.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:18.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:18.411 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:18.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:18.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:18.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:18.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:18.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:18.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:18.535 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:18.535 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:18.535 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:18.535 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:18.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:18.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-04-25T08:54:18.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T08:54:18.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:18.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:18.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-04-25T08:54:18.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:18.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:18.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:18.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:18.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:18.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:18.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:18.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-3 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:18.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:18.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T08:54:18.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:18.867 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T08:54:18.867 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T08:54:18.867 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:19.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:19.293 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:19.293 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:19.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:19.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:19.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:19.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:19.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:19.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:19.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:19.296 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:19.297 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:19.297 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:19.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:19.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:19.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:19.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:19.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:19.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:19.420 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:19.421 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:19.421 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:19.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:19.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:19.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-04-25T08:54:19.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T08:54:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-04-25T08:54:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:19.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:19.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:19.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:19.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:19.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T08:54:19.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:19.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:19.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-4 2024-04-25T08:54:19.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:19.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:19.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:19.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:19.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:19.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:19.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:19.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:19.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:19.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:19.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:19.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:19.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:19.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:19.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T08:54:19.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:19.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T08:54:19.770 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T08:54:19.770 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:20.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:20.219 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:20.219 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:20.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:20.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:20.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:20.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:20.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:20.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:20.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:20.221 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:20.222 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:20.222 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:20.222 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:20.222 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:20.223 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:20.223 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:20.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:20.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:20.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:20.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:20.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:20.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:20.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:20.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-04-25T08:54:20.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T08:54:20.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:20.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:20.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-04-25T08:54:20.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:20.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:20.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:20.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:20.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:20.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:20.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:20.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T08:54:20.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-5 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:20.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:20.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:20.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:20.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:20.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:20.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:20.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:20.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:20.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T08:54:20.659 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:21.074 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:21.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:21.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:21.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:21.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:21.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:21.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:21.078 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:21.078 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:21.078 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:21.078 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:21.079 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:21.079 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:21.079 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:21.079 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:21.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:21.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:21.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:21.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:21.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:21.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:21.187 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:21.187 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:21.188 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:21.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:21.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:21.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-04-25T08:54:21.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T08:54:21.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:21.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:21.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-04-25T08:54:21.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:21.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:21.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:21.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:21.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:21.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:21.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-6 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:21.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:21.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:21.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T08:54:21.509 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T08:54:21.509 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:21.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:21.949 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:21.949 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:21.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:21.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:21.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:21.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:21.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:22.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:22.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:22.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:22.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:22.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:22.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:22.072 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:22.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-04-25T08:54:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T08:54:22.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:22.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:22.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-04-25T08:54:22.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:22.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:22.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:22.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:22.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:22.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:22.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-7 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:22.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:22.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:22.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T08:54:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T08:54:22.538 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:22.943 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:22.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:22.946 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:22.947 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:22.947 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:22.947 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:22.947 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:22.947 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:23.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:23.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:23.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:23.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:23.065 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:23.065 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:23.065 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:23.065 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:23.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:23.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-04-25T08:54:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T08:54:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-04-25T08:54:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:23.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:23.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-8 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:23.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:23.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:23.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:23.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:23.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:23.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:23.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:23.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:23.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:23.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:23.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:23.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:23.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:23.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T08:54:23.458 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T08:54:23.458 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:23.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:23.860 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:23.860 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:23.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:23.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:23.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:23.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:23.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:23.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:23.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:23.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:23.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:23.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:23.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:23.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:23.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:23.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:23.863 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:23.863 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:23.863 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:23.863 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:23.863 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:23.864 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:23.864 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:23.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:23.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:23.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:23.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:23.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:23.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:23.985 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:23.986 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:23.986 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:23.986 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:23.986 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:23.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:23.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-04-25T08:54:24.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T08:54:24.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:24.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:24.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-04-25T08:54:24.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:24.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:24.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:24.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T08:54:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:24.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-9 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:24.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:24.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T08:54:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:24.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:24.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T08:54:24.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:24.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T08:54:24.369 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:24.369 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T08:54:24.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:24.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:24.807 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:24.808 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:24.808 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:24.808 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:24.808 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:24.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:24.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:24.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:24.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:24.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:24.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:24.927 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:24.927 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-04-25T08:54:25.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T08:54:25.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:25.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-04-25T08:54:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:25.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:25.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-10 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:25.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:25.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:25.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:25.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:25.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:25.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:25.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:25.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T08:54:25.273 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:25.273 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T08:54:25.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:25.786 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:25.786 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:25.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:25.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:25.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:25.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:25.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:25.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:25.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:25.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:25.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:25.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:25.789 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:25.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:25.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:25.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:25.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:25.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:25.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:25.805 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:25.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:25.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:25.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-04-25T08:54:25.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T08:54:25.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:25.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:25.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-04-25T08:54:25.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:25.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:25.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:25.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:25.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:25.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:25.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-11 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:54:25.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T08:54:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T08:54:26.134 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:26.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:26.584 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:26.584 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:26.584 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:26.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:26.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:26.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:26.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:26.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:26.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:26.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:26.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:26.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:26.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:26.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:26.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:26.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:26.587 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:26.587 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:26.587 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:26.587 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:26.588 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:26.588 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:26.588 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:26.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:26.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:26.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:26.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:26.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:26.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:26.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:26.706 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:26.706 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:26.706 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:26.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:26.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:26.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-04-25T08:54:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T08:54:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-04-25T08:54:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:26.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:26.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:26.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: get_pg pool-rep obj-warn-1 2024-04-25T08:54:26.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-04-25T08:54:26.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-warn-1 2024-04-25T08:54:26.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T08:54:26.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:54:27.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: local pgid=2.0 2024-04-25T08:54:27.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: get_osds pool-rep obj-warn-1 2024-04-25T08:54:27.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:27.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T08:54:27.304 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T08:54:27.304 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: object_osds=('3' '1' '0') 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: local object_osds 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:203: TEST_rados_repair_warning: local primary=3 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:204: TEST_rados_repair_warning: local bad_peer=1 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:206: TEST_rados_repair_warning: wait_for_clean 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:54:27.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:54:27.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:54:27.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:54:27.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:54:27.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:54:27.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:54:27.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:54:27.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:54:27.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:54:27.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:54:27.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:54:27.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:54:27.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:54:27.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:54:28.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:54:28.314 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:28.314 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:54:28.314 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:54:28.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:54:28.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:28.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:54:28.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803813 2024-04-25T08:54:28.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803813 2024-04-25T08:54:28.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813' 2024-04-25T08:54:28.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:28.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:54:28.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574881 2024-04-25T08:54:28.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574881 2024-04-25T08:54:28.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574881' 2024-04-25T08:54:28.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:28.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:54:28.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313243 2024-04-25T08:54:28.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313243 2024-04-25T08:54:28.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574881 2-90194313243' 2024-04-25T08:54:28.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:28.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:54:28.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953492 2024-04-25T08:54:28.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953492 2024-04-25T08:54:28.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574881 2-90194313243 3-137438953492' 2024-04-25T08:54:28.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:28.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803813 2024-04-25T08:54:28.954 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:28.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:54:28.955 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803813 2024-04-25T08:54:28.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:28.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803813 2024-04-25T08:54:28.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-04-25T08:54:28.957 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803813 2024-04-25T08:54:28.958 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:29.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803811 -lt 25769803813 2024-04-25T08:54:29.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:54:30.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:54:30.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:30.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803813 -lt 25769803813 2024-04-25T08:54:30.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:30.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:30.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574881 2024-04-25T08:54:30.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:54:30.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574881 2024-04-25T08:54:30.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:30.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574881 2024-04-25T08:54:30.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-04-25T08:54:30.784 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574881 2024-04-25T08:54:30.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:54:31.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574882 -lt 55834574881 2024-04-25T08:54:31.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:31.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313243 2024-04-25T08:54:31.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:31.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:54:31.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:31.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313243 2024-04-25T08:54:31.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313243 2024-04-25T08:54:31.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313243' 2024-04-25T08:54:31.252 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 90194313243 2024-04-25T08:54:31.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:54:31.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313243 -lt 90194313243 2024-04-25T08:54:31.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:31.655 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953492 2024-04-25T08:54:31.656 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:31.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:54:31.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953492 2024-04-25T08:54:31.658 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:31.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953492 2024-04-25T08:54:31.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953492' 2024-04-25T08:54:31.660 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 137438953492 2024-04-25T08:54:31.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:54:32.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953492 -lt 137438953492 2024-04-25T08:54:32.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:54:32.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:32.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:32.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:54:32.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:54:32.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:54:32.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:54:32.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:54:32.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:54:32.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:54:32.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:54:33.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:54:33.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:54:33.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:54:33.083 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:54:33.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:54:33.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:54:33.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:54:33.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: ceph pg 2.0 query 2024-04-25T08:54:33.643 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T08:54:33.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: COUNT=11 2024-04-25T08:54:33.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:208: TEST_rados_repair_warning: test 11 = 11 2024-04-25T08:54:33.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:209: TEST_rados_repair_warning: flush_pg_stats 2024-04-25T08:54:33.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:54:33.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:54:34.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:54:34.226 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:34.226 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:54:34.226 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:54:34.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:54:34.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:34.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:54:34.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803816 2024-04-25T08:54:34.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803816 2024-04-25T08:54:34.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803816' 2024-04-25T08:54:34.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:34.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:54:34.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574885 2024-04-25T08:54:34.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574885 2024-04-25T08:54:34.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803816 1-55834574885' 2024-04-25T08:54:34.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:34.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:54:34.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313246 2024-04-25T08:54:34.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313246 2024-04-25T08:54:34.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803816 1-55834574885 2-90194313246' 2024-04-25T08:54:34.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:34.656 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:54:34.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953495 2024-04-25T08:54:34.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953495 2024-04-25T08:54:34.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803816 1-55834574885 2-90194313246 3-137438953495' 2024-04-25T08:54:34.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:34.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803816 2024-04-25T08:54:34.793 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:34.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:54:34.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803816 2024-04-25T08:54:34.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:34.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803816 2024-04-25T08:54:34.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-04-25T08:54:34.796 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803816 2024-04-25T08:54:34.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:35.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803814 -lt 25769803816 2024-04-25T08:54:35.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:54:36.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:54:36.183 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:36.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803816 -lt 25769803816 2024-04-25T08:54:36.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:36.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574885 2024-04-25T08:54:36.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:36.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:54:36.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574885 2024-04-25T08:54:36.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:36.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574885 2024-04-25T08:54:36.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574885' 2024-04-25T08:54:36.582 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574885 2024-04-25T08:54:36.582 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:54:37.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574885 -lt 55834574885 2024-04-25T08:54:37.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:37.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313246 2024-04-25T08:54:37.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:54:37.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313246 2024-04-25T08:54:37.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:37.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313246 2024-04-25T08:54:37.022 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 90194313246 2024-04-25T08:54:37.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313246' 2024-04-25T08:54:37.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:54:37.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313246 -lt 90194313246 2024-04-25T08:54:37.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:37.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953495 2024-04-25T08:54:37.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:37.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:54:37.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953495 2024-04-25T08:54:37.452 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:37.453 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 137438953495 2024-04-25T08:54:37.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953495 2024-04-25T08:54:37.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953495' 2024-04-25T08:54:37.453 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:54:37.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953496 -lt 137438953495 2024-04-25T08:54:37.949 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-04-25T08:54:37.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T08:54:38.373 INFO:tasks.workunit.client.0.smithi088.stderr:dumped all 2024-04-25T08:54:38.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: COUNT=11 2024-04-25T08:54:38.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:211: TEST_rados_repair_warning: test 11 = 11 2024-04-25T08:54:38.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: ceph health 2024-04-25T08:54:38.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-04-25T08:54:38.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: grep -q 'osd.3 had 11 reads repaired' 2024-04-25T08:54:38.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: ceph health detail 2024-04-25T08:54:39.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:216: TEST_rados_repair_warning: ceph health mute OSD_TOO_MANY_REPAIRS 2024-04-25T08:54:40.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:217: TEST_rados_repair_warning: set -o pipefail 2024-04-25T08:54:40.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: ceph health 2024-04-25T08:54:40.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-04-25T08:54:40.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:220: TEST_rados_repair_warning: set +o pipefail 2024-04-25T08:54:40.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: seq 1 11 2024-04-25T08:54:40.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:40.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:40.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:40.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T08:54:40.661 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:40.661 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T08:54:41.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:41.072 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:41.072 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:41.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:41.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:41.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:41.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:41.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:41.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:41.073 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:41.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:41.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:41.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:41.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:41.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:41.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:41.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:41.076 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:41.076 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:41.077 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:41.077 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:41.077 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:41.077 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:41.077 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:41.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:41.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:41.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:41.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:41.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:41.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:41.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:41.185 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:41.185 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:41.185 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:41.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:41.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:41.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-04-25T08:54:41.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:41.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:41.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:41.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:41.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:41.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:41.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T08:54:41.303 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T08:54:41.303 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:41.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:41.703 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:41.703 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:41.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:41.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:41.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:41.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:41.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:41.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:41.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:41.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:41.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:41.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:41.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:41.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:41.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:41.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:41.707 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:41.707 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:41.707 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:41.707 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:41.708 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:41.708 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:41.708 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:41.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:41.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:41.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:41.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:41.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:41.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:41.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:41.832 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:41.832 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:41.832 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:41.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:41.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:41.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-1 2024-04-25T08:54:41.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:41.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:41.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:41.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:41.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:41.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:41.951 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:41.952 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:41.952 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:41.952 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:41.953 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:41.953 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:41.953 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:41.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:42.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:42.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T08:54:42.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:42.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:42.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-04-25T08:54:42.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:42.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:42.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:42.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:42.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:42.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:42.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:42.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:42.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:42.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:42.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T08:54:42.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:42.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T08:54:42.207 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T08:54:42.207 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:42.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:42.704 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:42.704 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:42.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:42.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:42.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:42.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:42.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:42.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:42.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:42.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:42.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:42.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:42.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:42.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:42.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:42.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:42.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:42.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:42.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:42.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:42.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:42.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:42.830 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:42.830 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:42.830 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:42.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:42.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:42.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-04-25T08:54:42.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:42.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:42.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:42.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T08:54:42.951 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T08:54:42.951 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:43.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:43.373 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:43.373 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:43.373 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:43.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:43.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:43.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:43.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:43.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:43.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:43.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:43.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:43.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:43.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:43.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:43.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:43.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:43.376 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:43.376 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:43.376 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:43.377 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:43.377 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:43.377 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:43.377 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:43.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:43.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:43.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:43.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:43.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:43.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:43.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:43.503 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:43.503 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:43.503 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:43.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:43.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:43.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-2 2024-04-25T08:54:43.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:43.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:43.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:43.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:43.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:43.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:43.624 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:43.625 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:43.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:43.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:43.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T08:54:43.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:43.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:43.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-04-25T08:54:43.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:43.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:43.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:43.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:43.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:43.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:43.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:43.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:43.805 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T08:54:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T08:54:43.806 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:44.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:44.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:44.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:44.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:44.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:44.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:44.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:44.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:44.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:44.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:44.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:44.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:44.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:44.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:44.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:44.356 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:44.356 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:44.356 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:44.356 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:44.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:44.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:44.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:44.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:44.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T08:54:44.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:44.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T08:54:44.466 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T08:54:44.466 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:44.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:44.854 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:44.854 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:44.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:44.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:44.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:44.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:44.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:44.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:44.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:44.856 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:44.857 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:44.857 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:44.857 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:44.857 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:44.858 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:44.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:44.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:44.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:44.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:44.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:44.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:44.980 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:44.980 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:44.980 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:44.980 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:44.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:44.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:44.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-3 2024-04-25T08:54:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:45.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:45.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:45.101 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:45.101 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:45.101 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:45.101 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:45.102 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:45.102 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:45.102 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:45.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:45.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:45.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:45.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:45.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:45.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T08:54:45.271 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T08:54:45.272 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:45.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:45.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:45.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:45.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:45.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:45.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:45.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:45.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:45.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:45.797 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:45.797 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:45.797 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:45.797 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:45.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:45.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:45.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-04-25T08:54:45.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:45.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:45.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T08:54:45.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:45.906 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T08:54:45.907 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T08:54:45.907 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:46.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:46.331 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:46.331 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:46.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:46.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:46.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:46.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:46.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:46.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:46.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:46.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:46.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:46.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:46.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:46.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:46.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:46.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:46.334 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:46.335 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:46.335 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:46.335 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:46.335 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:46.335 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:46.335 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:46.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:46.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:46.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:46.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:46.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:46.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:46.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:46.451 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:46.451 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:46.451 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:46.451 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:46.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:46.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-4 2024-04-25T08:54:46.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:46.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:46.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:46.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:46.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:46.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:46.571 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:46.571 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:46.571 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:46.571 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:46.571 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:46.572 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:46.572 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:46.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:46.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:46.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:46.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:46.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:46.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:46.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T08:54:46.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:46.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T08:54:46.753 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T08:54:46.753 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:47.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:47.142 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:47.142 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:47.142 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:47.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:47.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:47.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:47.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:47.143 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:47.145 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:47.146 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:47.146 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:47.146 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:47.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:47.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:47.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:47.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:47.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:47.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:47.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:47.253 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:47.253 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:47.253 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:47.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:47.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:47.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-04-25T08:54:47.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:47.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:47.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:47.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T08:54:47.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:47.363 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T08:54:47.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T08:54:47.363 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:47.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:47.792 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:47.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:47.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:47.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:47.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:47.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:47.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:47.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:47.796 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:47.796 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:47.796 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:47.796 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:47.796 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:47.796 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:47.796 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:47.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:47.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:47.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:47.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:47.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:47.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:47.904 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:47.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:47.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:47.905 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:47.905 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:47.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:47.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-5 2024-04-25T08:54:48.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:48.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:48.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:48.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:48.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:48.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:48.014 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:48.014 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:48.014 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:48.014 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:48.014 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:48.014 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:48.014 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:48.015 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:48.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:48.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:48.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:48.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:48.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:48.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:48.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T08:54:48.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:48.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T08:54:48.228 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T08:54:48.228 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:48.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:48.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:48.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:48.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:48.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:48.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:48.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:48.656 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:48.657 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:48.657 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:48.657 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:48.657 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:48.657 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:48.657 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:48.657 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:48.657 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:48.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:48.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:48.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:48.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:48.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:48.770 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:48.771 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:48.771 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:48.771 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:48.771 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:48.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:48.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-04-25T08:54:48.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:48.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:48.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:48.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:48.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:48.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:48.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:48.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T08:54:48.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:48.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:48.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:48.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:48.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:48.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T08:54:48.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:48.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T08:54:48.880 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T08:54:48.880 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:49.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:49.279 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:49.280 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:49.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:49.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:49.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:49.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:49.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:49.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:49.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:49.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:49.282 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:49.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:49.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:49.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:49.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:49.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:49.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:49.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:49.393 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:49.393 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:49.393 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:49.393 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:49.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:49.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-6 2024-04-25T08:54:49.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:49.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:49.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:49.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:49.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:49.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:49.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:49.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:49.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:49.512 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:49.512 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:49.512 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:49.512 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:49.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:49.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:49.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T08:54:49.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:49.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:49.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-04-25T08:54:49.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:49.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:49.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:49.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:49.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:49.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:49.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:49.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:49.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:49.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:49.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:49.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:49.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:49.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T08:54:49.774 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T08:54:49.774 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:50.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:50.253 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:50.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:50.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:50.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:50.255 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:50.256 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:50.256 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:50.256 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:50.256 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:50.256 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:50.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:50.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:50.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:50.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:50.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:50.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:50.365 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:50.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:50.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:50.366 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:50.366 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:50.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:50.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:50.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:50.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:50.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:50.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:50.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T08:54:50.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:50.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T08:54:50.490 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T08:54:50.491 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:50.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:50.900 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:50.901 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:50.901 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:50.901 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:50.901 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:50.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:51.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:51.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:51.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:51.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:51.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:51.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:51.011 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:51.011 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:51.011 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:51.011 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:51.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:51.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-7 2024-04-25T08:54:51.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:51.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:51.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:51.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:51.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:51.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:51.130 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:51.130 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:51.130 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:51.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:51.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:51.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T08:54:51.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:51.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:51.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-04-25T08:54:51.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:51.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:51.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:51.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:51.309 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:51.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:51.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:51.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:51.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:51.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T08:54:51.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:51.313 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T08:54:51.313 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T08:54:51.313 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:51.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:51.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:51.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:51.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:51.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:51.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:51.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:51.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:51.711 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:51.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:51.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:51.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:51.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:51.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:51.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:51.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:51.822 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:51.822 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:51.822 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:51.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:51.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:51.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-04-25T08:54:51.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:51.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:51.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:51.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:51.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:51.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:51.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:51.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T08:54:51.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:51.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:51.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:51.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:51.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:51.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T08:54:51.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:51.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T08:54:51.939 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:51.939 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T08:54:52.374 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:52.375 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:52.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:52.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:52.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:52.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:52.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:52.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:52.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:52.377 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:52.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:52.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:52.378 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:52.378 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:52.378 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:52.378 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:52.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:52.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:52.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:52.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:52.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:52.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:52.493 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:52.494 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:52.494 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:52.494 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:52.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:52.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:52.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-8 2024-04-25T08:54:52.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:52.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:52.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:52.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:52.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:52.602 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:52.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:52.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:52.603 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:52.604 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:52.604 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:52.604 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:52.604 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:52.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:52.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:52.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T08:54:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-04-25T08:54:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:52.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:52.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:52.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:52.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T08:54:52.778 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:53.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:53.165 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:53.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:53.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:53.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:53.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:53.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:53.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:53.167 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:53.167 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:53.167 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:53.167 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:53.167 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:53.167 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:53.167 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:53.167 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:53.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:53.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:53.280 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:53.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:53.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:53.281 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:53.281 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:53.281 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:53.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:53.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:53.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T08:54:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:53.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:53.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:53.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:53.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:53.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T08:54:53.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:53.391 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T08:54:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T08:54:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:53.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:53.850 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:53.850 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:53.850 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:53.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:53.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:53.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:53.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:53.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:53.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:53.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:53.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:53.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:53.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:53.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:53.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:53.858 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:53.859 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:53.859 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:53.859 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:53.859 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:53.859 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:53.859 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:53.859 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:53.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:53.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:53.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:53.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:53.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:53.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:53.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:53.969 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:53.969 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:53.969 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:53.969 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:53.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:53.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-9 2024-04-25T08:54:54.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:54.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:54.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:54.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:54.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:54.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:54.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:54.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:54.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:54.081 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:54.081 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:54.081 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:54.081 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:54.081 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:54.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:54.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:54.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:54.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:54.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:54.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T08:54:54.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:54.287 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T08:54:54.287 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T08:54:54.287 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:54.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:54.679 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:54.679 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:54.679 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:54.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:54.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:54.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:54.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:54.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:54.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:54.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:54.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:54.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:54.794 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:54.795 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:54.795 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:54.795 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:54.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:54.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:54.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-04-25T08:54:54.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:54.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:54.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:54.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:54.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:54.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T08:54:54.914 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T08:54:54.914 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:55.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:55.349 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:55.350 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:55.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:55.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:55.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:55.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:55.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:55.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:55.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:55.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:55.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:55.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:55.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:55.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:55.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:55.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:55.353 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:55.353 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:55.353 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:55.354 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:55.354 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:55.354 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:55.355 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:55.355 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:55.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:55.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:55.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:55.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:55.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:55.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:55.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:55.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-10 2024-04-25T08:54:55.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:55.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:55.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:55.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:55.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:55.586 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:55.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:55.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:55.586 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:55.587 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:55.587 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:55.587 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:55.587 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:55.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:55.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:55.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:55.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:55.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:54:55.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:55.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T08:54:55.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:55.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T08:54:55.752 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T08:54:55.752 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:56.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:56.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T08:54:56.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:56.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T08:54:56.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:56.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:56.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:56.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:56.152 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:56.152 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:56.152 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:56.152 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:56.152 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:56.152 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:56.153 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:56.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:56.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:56.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:56.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:56.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T08:54:56.266 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:56.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:56.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:56.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:56.267 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:56.267 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:56.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:56.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-04-25T08:54:56.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 1 2024-04-25T08:54:56.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:54:56.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:54:56.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:54:56.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:54:56.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T08:54:56.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:54:56.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T08:54:56.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:54:56.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:56.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:54:56.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:54:56.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:54:56.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T08:54:56.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T08:54:56.510 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T08:54:56.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T08:54:56.511 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:54:56.779 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T08:54:56.779 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:56.779 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:54:56.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T08:54:56.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T08:54:56.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:54:56.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:54:56.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:54:56.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:54:56.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:54:56.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:54:56.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:54:56.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:56.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:54:56.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:54:56.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:54:56.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:56.782 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:54:56.783 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:56.783 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:56.783 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:56.783 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:56.783 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:56.783 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:56.783 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:54:56.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:56.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:54:56.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:54:56.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:54:56.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:54:56.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:54:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:56.894 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:54:56.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:54:56.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr pool-rep obj-warn-11 2024-04-25T08:54:57.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T08:54:57.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:54:57.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T08:54:57.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T08:54:57.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T08:54:57.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:54:57.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T08:54:57.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T08:54:57.002 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T08:54:57.003 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:54:57.003 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:57.003 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:54:57.003 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.3.asok 2024-04-25T08:54:57.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T08:54:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:57.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:57.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T08:54:57.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:231: TEST_rados_repair_warning: wait_for_clean 2024-04-25T08:54:57.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:54:57.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:54:57.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:54:57.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:54:57.179 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:54:57.179 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:54:57.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:54:57.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:54:57.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:54:57.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:54:57.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:54:57.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:54:57.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:54:57.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:54:57.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:54:57.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:54:57.737 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:54:57.737 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:54:57.737 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:54:57.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:54:57.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:57.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:54:57.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803823 2024-04-25T08:54:57.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803823 2024-04-25T08:54:57.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823' 2024-04-25T08:54:57.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:57.890 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:54:58.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574891 2024-04-25T08:54:58.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574891 2024-04-25T08:54:58.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-55834574891' 2024-04-25T08:54:58.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:58.040 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:54:58.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313253 2024-04-25T08:54:58.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313253 2024-04-25T08:54:58.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-55834574891 2-90194313253' 2024-04-25T08:54:58.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:54:58.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:54:58.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953502 2024-04-25T08:54:58.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953502 2024-04-25T08:54:58.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-55834574891 2-90194313253 3-137438953502' 2024-04-25T08:54:58.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:54:58.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803823 2024-04-25T08:54:58.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:54:58.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:54:58.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803823 2024-04-25T08:54:58.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:54:58.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803823 2024-04-25T08:54:58.339 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803823 2024-04-25T08:54:58.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-04-25T08:54:58.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:54:58.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803821 -lt 25769803823 2024-04-25T08:54:58.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:54:59.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:54:59.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:55:00.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803823 -lt 25769803823 2024-04-25T08:55:00.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:00.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:00.261 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574891 2024-04-25T08:55:00.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:55:00.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574891 2024-04-25T08:55:00.263 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:00.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574891 2024-04-25T08:55:00.264 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574891 2024-04-25T08:55:00.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574891' 2024-04-25T08:55:00.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:55:00.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574892 -lt 55834574891 2024-04-25T08:55:00.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:00.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313253 2024-04-25T08:55:00.700 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:00.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:55:00.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313253 2024-04-25T08:55:00.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313253 2024-04-25T08:55:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313253' 2024-04-25T08:55:00.704 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 90194313253 2024-04-25T08:55:00.704 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:55:01.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313253 -lt 90194313253 2024-04-25T08:55:01.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:01.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953502 2024-04-25T08:55:01.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:01.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:55:01.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953502 2024-04-25T08:55:01.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:01.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953502 2024-04-25T08:55:01.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953502' 2024-04-25T08:55:01.156 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 137438953502 2024-04-25T08:55:01.156 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:55:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953502 -lt 137438953502 2024-04-25T08:55:01.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:55:01.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:01.603 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:02.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:55:02.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:55:02.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:55:02.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:55:02.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:55:02.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:55:02.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:55:02.159 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:55:02.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:55:02.533 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:55:02.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:02.534 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:03.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:55:03.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:55:03.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:55:03.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: ceph pg 2.0 query 2024-04-25T08:55:03.077 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T08:55:03.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: COUNT=22 2024-04-25T08:55:03.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: expr 11 '*' 2 2024-04-25T08:55:03.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: test 22 = 22 2024-04-25T08:55:03.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:234: TEST_rados_repair_warning: flush_pg_stats 2024-04-25T08:55:03.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:55:03.228 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:55:03.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:55:03.624 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:55:03.624 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T08:55:03.624 INFO:tasks.workunit.client.0.smithi088.stderr:3' 2024-04-25T08:55:03.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:55:03.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:03.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:55:03.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803826 2024-04-25T08:55:03.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803826 2024-04-25T08:55:03.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826' 2024-04-25T08:55:03.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:03.769 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:55:03.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574894 2024-04-25T08:55:03.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574894 2024-04-25T08:55:03.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-55834574894' 2024-04-25T08:55:03.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:03.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:55:04.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313256 2024-04-25T08:55:04.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313256 2024-04-25T08:55:04.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-55834574894 2-90194313256' 2024-04-25T08:55:04.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:04.052 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:55:04.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953505 2024-04-25T08:55:04.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953505 2024-04-25T08:55:04.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-55834574894 2-90194313256 3-137438953505' 2024-04-25T08:55:04.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:04.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803826 2024-04-25T08:55:04.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:04.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:55:04.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803826 2024-04-25T08:55:04.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:04.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803826 2024-04-25T08:55:04.222 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803826 2024-04-25T08:55:04.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-04-25T08:55:04.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:55:04.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803824 -lt 25769803826 2024-04-25T08:55:04.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:55:05.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:55:05.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:55:06.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803826 -lt 25769803826 2024-04-25T08:55:06.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:06.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574894 2024-04-25T08:55:06.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:06.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:55:06.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574894 2024-04-25T08:55:06.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:06.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574894 2024-04-25T08:55:06.025 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574894 2024-04-25T08:55:06.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574894' 2024-04-25T08:55:06.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:55:06.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574895 -lt 55834574894 2024-04-25T08:55:06.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:06.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313256 2024-04-25T08:55:06.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:06.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:55:06.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:06.470 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313256 2024-04-25T08:55:06.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313256 2024-04-25T08:55:06.471 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 90194313256 2024-04-25T08:55:06.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313256' 2024-04-25T08:55:06.471 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:55:06.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313256 -lt 90194313256 2024-04-25T08:55:06.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:06.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-137438953505 2024-04-25T08:55:06.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:06.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:55:06.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-137438953505 2024-04-25T08:55:06.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:06.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953505 2024-04-25T08:55:06.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 137438953505' 2024-04-25T08:55:06.922 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 137438953505 2024-04-25T08:55:06.923 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:55:07.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953505 -lt 137438953505 2024-04-25T08:55:07.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-04-25T08:55:07.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T08:55:07.735 INFO:tasks.workunit.client.0.smithi088.stderr:dumped all 2024-04-25T08:55:07.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: COUNT=33 2024-04-25T08:55:07.752 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: expr 11 '*' 3 2024-04-25T08:55:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: test 33 = 33 2024-04-25T08:55:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:240: TEST_rados_repair_warning: CHECKTIME=10 2024-04-25T08:55:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:241: TEST_rados_repair_warning: LOOPS=0 2024-04-25T08:55:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:242: TEST_rados_repair_warning: true 2024-04-25T08:55:07.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:244: TEST_rados_repair_warning: sleep 1 2024-04-25T08:55:08.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: ceph health 2024-04-25T08:55:08.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 2 OSDs' 2024-04-25T08:55:09.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:247: TEST_rados_repair_warning: break 2024-04-25T08:55:09.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: ceph health detail 2024-04-25T08:55:09.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: expr 11 '*' 2 2024-04-25T08:55:09.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: grep -q 'osd.3 had 22 reads repaired' 2024-04-25T08:55:09.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: ceph health detail 2024-04-25T08:55:09.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: grep -q 'osd.1 had 11 reads repaired' 2024-04-25T08:55:10.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:259: TEST_rados_repair_warning: delete_pool pool-rep 2024-04-25T08:55:10.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-04-25T08:55:10.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-04-25T08:55:10.942 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'pool-rep' does not exist 2024-04-25T08:55:10.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T08:55:10.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:10.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:55:10.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T08:55:10.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:55:10.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:55:10.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:55:10.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:55:10.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:55:11.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:55:11.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:55:11.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:55:11.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:55:11.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:55:11.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:55:11.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:55:11.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:11.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:55:11.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:55:11.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:11.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:55:11.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:55:11.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T08:55:11.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:55:11.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:11.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:11.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:55:11.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T08:55:11.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:55:11.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:55:11.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:55:11.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:55:11.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:55:11.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:55:11.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:55:11.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:55:11.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:55:11.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:55:11.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:55:11.320 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:55:11.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:11.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:55:11.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:55:11.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:55:11.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:55:11.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:55:11.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T08:55:11.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:55:11.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:11.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:11.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T08:55:11.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:55:11.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:55:11.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T08:55:11.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:55:11.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:11.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:11.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.389694 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:55:11.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T08:55:11.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T08:55:11.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:55:11.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:11.806 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:11.807 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:11.807 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:11.807 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:11.807 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:55:11.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T08:55:12.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:55:12.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:55:12.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:55:12.027 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:55:12.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:55:12.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:55:12.028 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:55:12.028 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:55:12.029 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:55:12.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:55:12.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:12.029 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:12.030 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-mon.a.asok 2024-04-25T08:55:12.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:55:12.030 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.389694/ceph-mon.a.asok config get fsid 2024-04-25T08:55:12.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:55:12.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:55:12.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:55:12.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:55:12.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:55:12.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:55:12.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:55:12.148 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:55:12.149 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:55:12.149 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:12.149 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:12.149 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-mon.a.asok 2024-04-25T08:55:12.149 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:55:12.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.389694/ceph-mon.a.asok config get mon_host 2024-04-25T08:55:12.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T08:55:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:55:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:55:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:55:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T08:55:12.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:55:13.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:55:13.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:13.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:13.176 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:13.176 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:13.176 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:13.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:55:13.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:55:13.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:55:13.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T08:55:13.529 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_backfill_unfound td/osd-rep-recov-eio.sh 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:264: TEST_rep_backfill_unfound: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:265: TEST_rep_backfill_unfound: local objname=myobject 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:266: TEST_rep_backfill_unfound: local lastobj=300 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:268: TEST_rep_backfill_unfound: local testobj=obj250 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:270: TEST_rep_backfill_unfound: export CEPH_ARGS 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:271: TEST_rep_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:272: TEST_rep_backfill_unfound: setup_osds 3 2024-04-25T08:55:13.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-04-25T08:55:13.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T08:55:13.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T08:55:13.548 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-04-25T08:55:13.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-04-25T08:55:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:55:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T08:55:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:55:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:55:13.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:55:13.554 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:55:13.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T08:55:13.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:55:13.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c76cf53c-5aa5-44a0-a9cf-c1edf4f5c1fa 2024-04-25T08:55:13.557 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 c76cf53c-5aa5-44a0-a9cf-c1edf4f5c1fa 2024-04-25T08:55:13.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c76cf53c-5aa5-44a0-a9cf-c1edf4f5c1fa' 2024-04-25T08:55:13.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:55:13.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxGipmdbUXIxAAyDIcAbpravZxoYn/nd84Og== 2024-04-25T08:55:13.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxGipmdbUXIxAAyDIcAbpravZxoYn/nd84Og=="}' 2024-04-25T08:55:13.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c76cf53c-5aa5-44a0-a9cf-c1edf4f5c1fa -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:55:13.922 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:55:13.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:55:13.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxGipmdbUXIxAAyDIcAbpravZxoYn/nd84Og== --osd-uuid c76cf53c-5aa5-44a0-a9cf-c1edf4f5c1fa 2024-04-25T08:55:13.976 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:13.982+0000 7f075431c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:13.999 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:14.006+0000 7f075431c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:14.000 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:14.006+0000 7f075431c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:14.001 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:14.006+0000 7f075431c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:55:14.001 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:14.006+0000 7f075431c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:55:14.001 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:14.006+0000 7f075431c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:55:14.039 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:14.046+0000 7f075431c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T08:55:20.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T08:55:20.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:55:20.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:55:20.528 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:55:20.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:55:21.083 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:55:21.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:55:21.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:55:21.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:55:21.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:55:21.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:55:21.120 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:21.130+0000 7f9d050847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:21.173 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:21.186+0000 7f9d050847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:21.174 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:21.186+0000 7f9d050847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:21.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:55:21.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:55:21.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:55:21.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:55:21.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:55:21.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:21.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:55:21.491 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:55:21.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:21.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:21.740 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:21.750+0000 7f9d050847c0 -1 Falling back to public interface 2024-04-25T08:55:21.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:22.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:22.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:22.937 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:55:22.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:55:22.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:22.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:23.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:24.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:24.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:24.342 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:55:24.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:55:24.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:24.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:24.343 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:24.354+0000 7f9d050847c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:55:24.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:25.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:25.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:25.817 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:55:25.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:55:25.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:25.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:26.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:27.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:27.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:27.264 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:55:27.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:55:27.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:27.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:27.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:28.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:28.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:28.701 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:55:28.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:55:28.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:28.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:29.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:30.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:30.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:30.143 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:55:30.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:55:30.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:30.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:30.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:31.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:31.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:31.570 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:55:31.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:55:31.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:31.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:32.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:33.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:33.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:33.156 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:55:33.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:55:33.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:33.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:33.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:34.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:34.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:34.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:55:34.583 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:55:34.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:34.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:35.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:36.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:36.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:36.193 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:55:36.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:55:36.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:36.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:36.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:37.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:37.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:37.636 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:55:37.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:55:37.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:37.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:38.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:39.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:39.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:39.363 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:55:39.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:55:39.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:39.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:40.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:41.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:41.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:55:41.241 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:55:41.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:41.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:42.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:43.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:43.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:43.014 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:55:43.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:55:43.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:43.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:44.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:45.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:45.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:45.223 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:55:45.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:55:45.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:45.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:45.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:46.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:46.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:46.988 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:55:46.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:55:46.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:46.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:48.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:49.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:49.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:49.017 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:55:49.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:55:49.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:49.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:55:49.594 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1503545333,v1:127.0.0.1:6803/1503545333] [v2:127.0.0.1:6804/1503545333,v1:127.0.0.1:6805/1503545333] exists,up c76cf53c-5aa5-44a0-a9cf-c1edf4f5c1fa 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T08:55:49.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:55:49.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T08:55:49.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:55:49.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3fc9eadf-5689-4093-83dc-1368fcb08a32 2024-04-25T08:55:49.601 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 3fc9eadf-5689-4093-83dc-1368fcb08a32 2024-04-25T08:55:49.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3fc9eadf-5689-4093-83dc-1368fcb08a32' 2024-04-25T08:55:49.601 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:55:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVGipmEurOJRAAjelteDVqfGMLMGJrVH8uFQ== 2024-04-25T08:55:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVGipmEurOJRAAjelteDVqfGMLMGJrVH8uFQ=="}' 2024-04-25T08:55:49.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3fc9eadf-5689-4093-83dc-1368fcb08a32 -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:55:50.287 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:55:50.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:55:50.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVGipmEurOJRAAjelteDVqfGMLMGJrVH8uFQ== --osd-uuid 3fc9eadf-5689-4093-83dc-1368fcb08a32 2024-04-25T08:55:50.348 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:50.357+0000 7fa06a3ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:50.357 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:50.369+0000 7fa06a3ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:50.358 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:50.369+0000 7fa06a3ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:50.359 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:50.369+0000 7fa06a3ca7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:55:50.359 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:50.369+0000 7fa06a3ca7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:55:50.359 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:50.369+0000 7fa06a3ca7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:55:50.388 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:50.397+0000 7fa06a3ca7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T08:55:57.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T08:55:57.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:55:57.190 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:55:57.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:55:57.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:55:57.792 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:55:57.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:55:57.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:55:57.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:55:57.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:55:57.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:55:57.826 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:57.837+0000 7ffb1e8077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:57.828 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:57.837+0000 7ffb1e8077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:57.829 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:57.841+0000 7ffb1e8077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:58.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:55:58.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:55:58.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:55:58.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:55:58.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:55:58.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:58.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:55:58.246 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:55:58.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:58.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:55:58.390 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:55:58.401+0000 7ffb1e8077c0 -1 Falling back to public interface 2024-04-25T08:55:58.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:59.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:59.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:59.655 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:55:59.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:55:59.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:59.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:00.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:01.017 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:01.029+0000 7ffb1e8077c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:56:01.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:01.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:01.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:56:01.132 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:56:01.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:01.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:01.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:02.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:02.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:02.776 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:56:02.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:56:02.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:02.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:03.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:04.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:04.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:04.278 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:56:04.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:56:04.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:04.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:04.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:05.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:05.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:05.734 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:56:05.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:56:05.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:05.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:06.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:07.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:07.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:07.299 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:07.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:08.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:08.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:08.719 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:56:08.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:56:08.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:08.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:09.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:10.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:10.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:10.138 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:56:10.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:56:10.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:10.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:10.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:11.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:11.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:11.672 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:56:11.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:56:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:12.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:13.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:13.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:13.362 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:56:13.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:56:13.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:13.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:14.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:15.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:15.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:15.036 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:56:15.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:56:15.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:15.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:15.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:16.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:16.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:16.691 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:56:16.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:56:16.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:16.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:17.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:18.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:18.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:18.199 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:56:18.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:56:18.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:18.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:19.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:20.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:20.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:20.092 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:56:20.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:56:20.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:20.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:20.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:21.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:21.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:21.882 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:56:21.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:56:21.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:21.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:22.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:23.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:23.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:23.348 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:56:23.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:56:23.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:23.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:24.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:25.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:25.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:25.475 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:56:25.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:56:25.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:25.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:25.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:26.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:26.950 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T08:56:26.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:26.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:56:26.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:26.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:27.469 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3517336327,v1:127.0.0.1:6811/3517336327] [v2:127.0.0.1:6812/3517336327,v1:127.0.0.1:6813/3517336327] exists,up 3fc9eadf-5689-4093-83dc-1368fcb08a32 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T08:56:27.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:56:27.474 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:56:27.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:56:27.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T08:56:27.476 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:56:27.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f52fafd0-bebf-4d35-9610-ce771c101535 2024-04-25T08:56:27.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f52fafd0-bebf-4d35-9610-ce771c101535' 2024-04-25T08:56:27.477 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 f52fafd0-bebf-4d35-9610-ce771c101535 2024-04-25T08:56:27.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:56:27.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7GipmCiRwHhAAzO9AyGDNDL/+e9IJEpbF2g== 2024-04-25T08:56:27.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7GipmCiRwHhAAzO9AyGDNDL/+e9IJEpbF2g=="}' 2024-04-25T08:56:27.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f52fafd0-bebf-4d35-9610-ce771c101535 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T08:56:28.028 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:56:28.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T08:56:28.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7GipmCiRwHhAAzO9AyGDNDL/+e9IJEpbF2g== --osd-uuid f52fafd0-bebf-4d35-9610-ce771c101535 2024-04-25T08:56:28.075 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:28.084+0000 7fe68b4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:28.175 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:28.184+0000 7fe68b4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:28.177 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:28.188+0000 7fe68b4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:28.178 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:28.188+0000 7fe68b4947c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:56:28.178 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:28.188+0000 7fe68b4947c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:56:28.178 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:28.188+0000 7fe68b4947c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T08:56:28.212 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:28.220+0000 7fe68b4947c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T08:56:34.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T08:56:34.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:56:34.976 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T08:56:34.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:56:34.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:56:35.547 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T08:56:35.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:56:35.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:56:35.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:56:35.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:56:35.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:56:35.580 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:35.592+0000 7fbee1a4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:35.583 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:35.592+0000 7fbee1a4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:35.584 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:35.592+0000 7fbee1a4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:35.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:56:35.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:56:35.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:56:35.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:56:35.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:56:35.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:35.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:56:35.972 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:56:35.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:35.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:36.118 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:36.128+0000 7fbee1a4f7c0 -1 Falling back to public interface 2024-04-25T08:56:36.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:37.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:37.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:37.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:56:37.450 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:56:37.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:37.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:37.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:38.808 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:56:38.816+0000 7fbee1a4f7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:56:38.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:38.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:38.942 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:56:38.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:56:38.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:38.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:39.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:40.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:40.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:40.590 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:56:40.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:56:40.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:40.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:41.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:42.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:42.148 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:56:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:56:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:42.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:42.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:43.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:43.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:43.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:56:43.722 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:56:43.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:43.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:44.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:45.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:45.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:45.156 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:56:45.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:56:45.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:45.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:45.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:46.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:46.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:46.650 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:56:46.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:56:46.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:46.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:47.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:48.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:48.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:48.236 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:56:48.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:56:48.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:48.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:48.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:49.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:49.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:49.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:56:49.735 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:56:49.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:49.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:50.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:51.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:51.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:51.159 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:56:51.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:56:51.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:51.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:51.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:52.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:52.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:52.710 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:56:52.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:56:52.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:52.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:53.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:54.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:54.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:54.691 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:56:54.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:56:54.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:54.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:55.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:56.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:56.340 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:56:56.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:56:56.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:56.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:57.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:58.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:58.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:58.138 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:56:58.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:56:58.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:58.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:56:59.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:00.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:00.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:00.370 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:57:00.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:57:00.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:00.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:01.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:02.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:02.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:02.189 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:57:02.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:57:02.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:02.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:03.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:04.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:04.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:04.245 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:57:04.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:57:04.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:04.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:04.703 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3964091322,v1:127.0.0.1:6819/3964091322] [v2:127.0.0.1:6820/3964091322,v1:127.0.0.1:6821/3964091322] exists,up f52fafd0-bebf-4d35-9610-ce771c101535 2024-04-25T08:57:04.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:57:04.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:57:04.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:57:04.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:57:04.705 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:57:04.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:57:04.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:57:04.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:57:04.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:57:04.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:57:04.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:57:05.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:57:05.334 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:57:05.334 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T08:57:05.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:57:05.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:05.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:57:05.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803800 2024-04-25T08:57:05.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803800 2024-04-25T08:57:05.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800' 2024-04-25T08:57:05.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:05.498 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:57:05.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574862 2024-04-25T08:57:05.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574862 2024-04-25T08:57:05.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800 1-55834574862' 2024-04-25T08:57:05.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:05.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:57:05.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345930 2024-04-25T08:57:05.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345930 2024-04-25T08:57:05.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800 1-55834574862 2-85899345930' 2024-04-25T08:57:05.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:05.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803800 2024-04-25T08:57:05.981 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:05.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:57:05.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803800 2024-04-25T08:57:05.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:05.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803800 2024-04-25T08:57:05.984 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803800 2024-04-25T08:57:05.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803800' 2024-04-25T08:57:05.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:57:06.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803800 -lt 25769803800 2024-04-25T08:57:06.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:06.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574862 2024-04-25T08:57:06.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:06.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:57:06.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574862 2024-04-25T08:57:06.637 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:06.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574862 2024-04-25T08:57:06.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574862' 2024-04-25T08:57:06.639 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574862 2024-04-25T08:57:06.639 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:07.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574860 -lt 55834574862 2024-04-25T08:57:07.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:57:08.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:57:08.046 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:08.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574863 -lt 55834574862 2024-04-25T08:57:08.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:08.501 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:08.502 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345930 2024-04-25T08:57:08.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:57:08.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345930 2024-04-25T08:57:08.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:08.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345930 2024-04-25T08:57:08.505 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345930 2024-04-25T08:57:08.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345930' 2024-04-25T08:57:08.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:57:08.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345930 -lt 85899345930 2024-04-25T08:57:08.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:57:08.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:08.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:09.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T08:57:09.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:57:09.551 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:57:09.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:57:09.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:57:09.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:57:09.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:57:09.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:57:09.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-25T08:57:09.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:57:09.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:09.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:10.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 8 2024-04-25T08:57:10.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 6 '!=' -1 2024-04-25T08:57:10.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:57:10.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=6 2024-04-25T08:57:10.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:57:10.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:57:10.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:57:10.491 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:57:10.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:57:10.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:57:10.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:57:10.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:57:10.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:57:10.492 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:57:10.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-25T08:57:10.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:57:10.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:10.913 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 8 2024-04-25T08:57:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 6 '!=' 6 2024-04-25T08:57:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:57:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:57:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:57:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:57:11.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:57:11.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:57:11.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:57:11.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:57:11.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:57:11.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T08:57:11.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:57:11.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T08:57:12.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:57:12.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:57:12.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:57:12.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:57:12.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:57:12.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:57:12.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:57:12.181 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:57:12.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:57:12.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:57:12.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:12.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:13.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T08:57:13.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:57:13.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:57:13.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:274: TEST_rep_backfill_unfound: local poolname=test-pool 2024-04-25T08:57:13.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:275: TEST_rep_backfill_unfound: create_pool test-pool 1 1 2024-04-25T08:57:13.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-04-25T08:57:13.877 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test-pool' already exists 2024-04-25T08:57:13.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:57:14.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:276: TEST_rep_backfill_unfound: wait_for_clean 2024-04-25T08:57:14.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:57:14.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:57:14.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:57:14.897 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:57:14.897 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:57:14.898 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:57:14.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:57:14.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:57:14.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:57:15.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:57:15.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:57:15.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:57:15.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:57:15.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:57:15.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:57:15.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:57:15.450 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:57:15.450 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T08:57:15.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:57:15.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:15.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:57:15.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803804 2024-04-25T08:57:15.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803804 2024-04-25T08:57:15.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804' 2024-04-25T08:57:15.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:15.648 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:57:15.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574866 2024-04-25T08:57:15.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574866 2024-04-25T08:57:15.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-55834574866' 2024-04-25T08:57:15.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:15.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:57:16.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345934 2024-04-25T08:57:16.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345934 2024-04-25T08:57:16.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-55834574866 2-85899345934' 2024-04-25T08:57:16.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:16.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803804 2024-04-25T08:57:16.033 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:16.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:57:16.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803804 2024-04-25T08:57:16.034 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:16.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803804 2024-04-25T08:57:16.035 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803804 2024-04-25T08:57:16.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2024-04-25T08:57:16.036 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:57:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803804 -lt 25769803804 2024-04-25T08:57:16.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:16.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574866 2024-04-25T08:57:16.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:16.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:57:16.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574866 2024-04-25T08:57:16.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:16.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574866 2024-04-25T08:57:16.481 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574866 2024-04-25T08:57:16.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574866' 2024-04-25T08:57:16.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:16.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574864 -lt 55834574866 2024-04-25T08:57:16.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:57:17.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:57:17.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:18.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574867 -lt 55834574866 2024-04-25T08:57:18.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:18.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345934 2024-04-25T08:57:18.301 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:18.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:57:18.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345934 2024-04-25T08:57:18.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:18.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345934 2024-04-25T08:57:18.305 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345934 2024-04-25T08:57:18.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345934' 2024-04-25T08:57:18.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:57:18.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345934 -lt 85899345934 2024-04-25T08:57:18.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:57:18.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:18.762 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:19.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:57:19.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:57:19.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:57:19.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:57:19.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:57:19.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:57:19.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:57:19.315 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:57:19.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:57:19.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:57:19.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:19.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:20.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:57:20.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:57:20.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:57:20.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:278: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T08:57:20.673 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.330675+0000 0'0 26:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:00:38.531308+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.331394+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:25:17.131493+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:07.221234+0000 0'0 25:16 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:25:34.330244+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.330773+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:00:28.596608+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:15.531899+0000 0'0 25:10 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T08:57:13.602588+0000 0'0 2024-04-25T08:57:13.602588+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:02:53.414558+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.331138+0000 0'0 26:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:57:51.389052+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:07.221291+0000 0'0 26:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:55:20.623775+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:07.221947+0000 0'0 25:16 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:21:05.518045+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.331408+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:25:15.031397+0000 0 0 2024-04-25T08:57:20.674 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:57:20.675 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:57:20.675 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T08:57:20.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:280: TEST_rep_backfill_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-04-25T08:57:20.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:57:20.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-04-25T08:57:20.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-04-25T08:57:20.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:57:20.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T08:57:20.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:57:20.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T08:57:20.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:57:20.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T08:57:20.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T08:57:20.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T08:57:20.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:20.851 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: get_osds test-pool myobject 2024-04-25T08:57:20.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T08:57:20.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-25T08:57:20.852 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-04-25T08:57:20.852 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:57:21.253 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: initial_osds=('2' '1' '0') 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: local -a initial_osds 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:283: TEST_rep_backfill_unfound: local last_osd=0 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:284: TEST_rep_backfill_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:57:21.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:57:21.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:57:21.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:57:21.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:285: TEST_rep_backfill_unfound: ceph osd down 0 2024-04-25T08:57:22.565 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T08:57:22.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:286: TEST_rep_backfill_unfound: ceph osd out 0 2024-04-25T08:57:23.254 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T08:57:23.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:288: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.330675+0000 0'0 26:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:00:38.531308+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.331394+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:25:17.131493+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:07.221234+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:25:34.330244+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.330773+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:00:28.596608+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:15.531899+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T08:57:13.602588+0000 0'0 2024-04-25T08:57:13.602588+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:29:41.821206+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.331138+0000 0'0 26:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:57:51.389052+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-04-25T08:57:07.221291+0000 0'0 26:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:55:20.623775+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:07.221947+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:21:05.518045+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:57:06.331408+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:25:15.031397+0000 0 0 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:57:23.669 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:57:23.670 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T08:57:23.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:290: TEST_rep_backfill_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-04-25T08:57:23.685 INFO:tasks.workunit.client.0.smithi088.stderr:4+0 records in 2024-04-25T08:57:23.685 INFO:tasks.workunit.client.0.smithi088.stderr:4+0 records out 2024-04-25T08:57:23.685 INFO:tasks.workunit.client.0.smithi088.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000169411 s, 24.2 MB/s 2024-04-25T08:57:23.686 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: seq 1 300 2024-04-25T08:57:23.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:23.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:23.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:23.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:24.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:24.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:24.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:24.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:24.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:24.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:24.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:24.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:24.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:24.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:24.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:24.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:25.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:25.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:25.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:25.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:25.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:25.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:25.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:25.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:25.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:25.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:25.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:25.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:25.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:25.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:26.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:26.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:26.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:26.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:26.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:26.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:26.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:26.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:26.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:26.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:26.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:26.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:27.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:27.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:27.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:27.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:27.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:27.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:27.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:27.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:27.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:27.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:27.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:27.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:28.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:28.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:28.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:28.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:28.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:28.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:28.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:28.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:28.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:28.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:28.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:28.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:29.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:29.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:29.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:29.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:29.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:29.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:29.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:29.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:29.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:29.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:29.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:29.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:29.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:29.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:30.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:30.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:30.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:30.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:30.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:30.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:30.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:30.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:30.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:30.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:30.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:31.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:31.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:31.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:31.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:31.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:31.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:31.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:31.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:31.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:31.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:31.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:32.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:32.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:32.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:32.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:32.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:32.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:32.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:32.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:32.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:32.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:32.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:32.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:33.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:33.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:33.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:33.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:33.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:33.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:33.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:33.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:33.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:33.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:33.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:33.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:33.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:33.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:34.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:34.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:34.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:34.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:34.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:34.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:34.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:34.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:34.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:34.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:34.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:35.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:35.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:35.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:35.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:35.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:35.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:35.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:35.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:35.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:35.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:35.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:35.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:36.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:36.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:36.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:36.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:36.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:36.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:36.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:36.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:36.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:36.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:36.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:37.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:37.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:37.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:37.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:37.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:37.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:37.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:37.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:37.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:37.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:37.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:37.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:37.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:37.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:38.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:38.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:38.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:38.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:38.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:38.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:38.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:38.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:38.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:38.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:39.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:39.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:39.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:39.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:39.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:39.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:39.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:39.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:39.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:39.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:39.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:39.796 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj101 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:39.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:39.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj102 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:40.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:40.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj103 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:40.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:40.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj104 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:40.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:40.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj105 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:40.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:40.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj106 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:40.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:40.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj107 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:40.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:40.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj108 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:41.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:41.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj109 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:41.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:41.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj110 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:41.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:41.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj111 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:41.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:41.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj112 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:41.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:41.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj113 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:41.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:41.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj114 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:42.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:42.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj115 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:42.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:42.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj116 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:42.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:42.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj117 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:42.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:42.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj118 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:42.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:42.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj119 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:42.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:42.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj120 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:42.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:42.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj121 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:43.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:43.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj122 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:43.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:43.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj123 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:43.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:43.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj124 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:43.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:43.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj125 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:43.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:43.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj126 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:43.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:43.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj127 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:44.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:44.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj128 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:44.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:44.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj129 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:44.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:44.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj130 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:44.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:44.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj131 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:44.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:44.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj132 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:45.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:45.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj133 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:45.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:45.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj134 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:45.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:45.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj135 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:45.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:45.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj136 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:45.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:45.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj137 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:45.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:45.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj138 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:46.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:46.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj139 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:46.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj140 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:46.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:46.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj141 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:46.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:46.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj142 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:46.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:46.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj143 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:46.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:46.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj144 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:47.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:47.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj145 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:47.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:47.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj146 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:47.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:47.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj147 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:47.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:47.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj148 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:47.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:47.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj149 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:47.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:47.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj150 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:47.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:47.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj151 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:48.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:48.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj152 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:48.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:48.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj153 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:48.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:48.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj154 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:48.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:48.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj155 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:48.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:48.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj156 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:49.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:49.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj157 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:49.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:49.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj158 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:49.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:49.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj159 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:49.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:49.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj160 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:49.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:49.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj161 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:49.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:49.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj162 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:49.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:49.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj163 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:50.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:50.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj164 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:50.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:50.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj165 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:50.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:50.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj166 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:50.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:50.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj167 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:50.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:50.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj168 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:50.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:50.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj169 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:51.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:51.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj170 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:51.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:51.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj171 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:51.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:51.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj172 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:51.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:51.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj173 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:51.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj174 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:51.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:51.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj175 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:52.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:52.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj176 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:52.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:52.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj177 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:52.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:52.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj178 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:52.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:52.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj179 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:52.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:52.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj180 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:52.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:52.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj181 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:53.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:53.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj182 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:53.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:53.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj183 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:53.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:53.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj184 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj185 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj186 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj187 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:53.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:53.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj188 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:54.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:54.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj189 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:54.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:54.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj190 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:54.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:54.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj191 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:54.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:54.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj192 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:54.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:54.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj193 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:54.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:54.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj194 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:55.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:55.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj195 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:55.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:55.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj196 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:55.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:55.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj197 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:55.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:55.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj198 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:55.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:55.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj199 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:55.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:55.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj200 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:56.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:56.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj201 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:56.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:56.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj202 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:56.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:56.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj203 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:56.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:56.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj204 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:56.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:56.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj205 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:56.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:56.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj206 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:57.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:57.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj207 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:57.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:57.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj208 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:57.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:57.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj209 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:57.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:57.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj210 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:57.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:57.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj211 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:57.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj212 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:57.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:57.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj213 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:58.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:58.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj214 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:58.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:58.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj215 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:58.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:58.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj216 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:58.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:58.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj217 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:58.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:58.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj218 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:58.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:58.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj219 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:59.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:59.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj220 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:59.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:59.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj221 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:59.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:59.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj222 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:59.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj223 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:59.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:59.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj224 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:57:59.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:57:59.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj225 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:00.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:00.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj226 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:00.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj227 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:00.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:00.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj228 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:00.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:00.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj229 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:00.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:00.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj230 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:00.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:00.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj231 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:01.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:01.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj232 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:01.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:01.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj233 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:01.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:01.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj234 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:01.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:01.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj235 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:01.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:01.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj236 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:01.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:01.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj237 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:01.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:01.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj238 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:02.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:02.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj239 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:02.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:02.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj240 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:02.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:02.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj241 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:02.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:02.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj242 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:02.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:02.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj243 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:02.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:02.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj244 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:03.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:03.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj245 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:03.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:03.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj246 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:03.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:03.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj247 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:03.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:03.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj248 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:03.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:03.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj249 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:03.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:03.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj250 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:04.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:04.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj251 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:04.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:04.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj252 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:04.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:04.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj253 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:04.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:04.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj254 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:04.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:04.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj255 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:04.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:04.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj256 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj257 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:05.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:05.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj258 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:05.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:05.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj259 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:05.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:05.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj260 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:05.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:05.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj261 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:05.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:05.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj262 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:05.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:05.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj263 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:06.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:06.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj264 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:06.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:06.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj265 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:06.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:06.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj266 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:06.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:06.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj267 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:06.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:06.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj268 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:06.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:06.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj269 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:07.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:07.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj270 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:07.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:07.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj271 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:07.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:07.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj272 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:07.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:07.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj273 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:07.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:07.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj274 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:07.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:07.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj275 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:08.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:08.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj276 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:08.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:08.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj277 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:08.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:08.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj278 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:08.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:08.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj279 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:08.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:08.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj280 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:08.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:08.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj281 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:09.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:09.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj282 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:09.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:09.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj283 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:09.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:09.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj284 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:09.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:09.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj285 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:09.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:09.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj286 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:09.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:09.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj287 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:09.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:09.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj288 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:10.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:10.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj289 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:10.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:10.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj290 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:10.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:10.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj291 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:10.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:10.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj292 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:10.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:10.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj293 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:10.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:10.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj294 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:11.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:11.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj295 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:11.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:11.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj296 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:11.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:11.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj297 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:11.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:11.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj298 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:11.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:11.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj299 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:11.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:11.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj300 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T08:58:12.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:296: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 0 2024-04-25T08:58:12.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:12.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:12.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj250 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj250 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T08:58:12.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-25T08:58:12.019 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-04-25T08:58:12.019 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:12.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T08:58:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T08:58:12.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T08:58:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T08:58:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=2 2024-04-25T08:58:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:12.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-04-25T08:58:12.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:12.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-04-25T08:58:12.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:12.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-04-25T08:58:12.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:12.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:12.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:12.461 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-04-25T08:58:12.461 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T08:58:12.461 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T08:58:12.462 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:12.462 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:12.462 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:58:12.462 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.2.asok 2024-04-25T08:58:12.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:12.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:12.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:12.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:12.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.2 2024-04-25T08:58:12.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T08:58:12.587 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T08:58:12.587 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:12.587 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:12.587 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:58:12.588 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.2.asok 2024-04-25T08:58:12.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:12.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.2.asok injectdataerr test-pool obj250 2024-04-25T08:58:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:297: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 1 2024-04-25T08:58:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T08:58:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T08:58:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T08:58:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T08:58:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T08:58:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T08:58:12.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj250 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj250 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-25T08:58:12.698 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:58:12.699 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-04-25T08:58:13.125 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T08:58:13.126 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T08:58:13.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T08:58:13.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T08:58:13.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T08:58:13.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T08:58:13.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T08:58:13.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T08:58:13.126 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T08:58:13.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T08:58:13.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T08:58:13.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T08:58:13.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T08:58:13.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T08:58:13.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T08:58:13.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T08:58:13.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T08:58:13.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:58:13.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:58:13.129 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:13.129 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:13.129 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:58:13.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:58:13.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T08:58:13.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T08:58:13.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T08:58:13.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T08:58:13.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T08:58:13.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:58:13.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:58:13.239 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:13.239 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:13.239 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:58:13.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T08:58:13.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T08:58:13.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr test-pool obj250 2024-04-25T08:58:13.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:299: TEST_rep_backfill_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-04-25T08:58:13.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:58:13.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:58:13.349 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:58:13.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:58:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:58:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:58:13.351 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:58:13.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:58:13.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:58:13.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:58:13.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:58:13.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:58:13.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:58:13.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:58:13.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:58:13.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:58:13.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:58:13.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T08:58:13.352 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:58:13.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T08:58:13.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:58:13.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-04-25T08:58:13.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T08:58:13.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:58:13.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:58:13.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:58:13.389 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:58:13.398+0000 7fc7f3aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:13.391 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:58:13.398+0000 7fc7f3aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:13.392 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:58:13.402+0000 7fc7f3aff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:13.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T08:58:13.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:58:13.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:58:13.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:58:13.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:58:13.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:13.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:58:13.738 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:58:13.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:13.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:58:13.950 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:58:13.958+0000 7fc7f3aff7c0 -1 Falling back to public interface 2024-04-25T08:58:14.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:15.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:15.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:15.185 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:58:15.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:58:15.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:15.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:58:15.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:16.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:16.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:16.608 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:58:16.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:58:16.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:16.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:58:17.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:17.093 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:58:17.102+0000 7fc7f3aff7c0 -1 osd.0 26 log_to_monitors true 2024-04-25T08:58:18.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:18.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:18.025 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:58:18.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:58:18.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:18.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:58:18.779 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up out weight 0 up_from 32 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/600823897,v1:127.0.0.1:6803/600823897] [v2:127.0.0.1:6804/600823897,v1:127.0.0.1:6805/600823897] exists,up c76cf53c-5aa5-44a0-a9cf-c1edf4f5c1fa 2024-04-25T08:58:18.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:58:18.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:58:18.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:58:18.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:300: TEST_rep_backfill_unfound: ceph osd in 0 2024-04-25T08:58:20.280 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already in. 2024-04-25T08:58:20.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:302: TEST_rep_backfill_unfound: sleep 15 2024-04-25T08:58:35.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: seq 1 360 2024-04-25T08:58:35.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T08:58:35.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T08:58:35.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T08:58:35.304 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T08:58:35.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T08:58:35.305 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T08:58:35.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:35.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:35.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T08:58:35.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T08:58:35.719 INFO:tasks.workunit.client.0.smithi088.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:35.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T08:58:35.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T08:58:36.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T08:58:36.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T08:58:36.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T08:58:36.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T08:58:36.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T08:58:36.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T08:58:37.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:37.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:37.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T08:58:37.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T08:58:37.145 INFO:tasks.workunit.client.0.smithi088.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:37.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T08:58:37.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T08:58:38.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T08:58:38.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T08:58:38.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T08:58:38.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T08:58:38.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T08:58:38.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T08:58:38.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:38.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:38.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T08:58:38.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T08:58:38.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T08:58:38.575 INFO:tasks.workunit.client.0.smithi088.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:38.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T08:58:39.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T08:58:39.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T08:58:39.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T08:58:39.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T08:58:39.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T08:58:39.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T08:58:40.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:40.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:40.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T08:58:40.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T08:58:40.015 INFO:tasks.workunit.client.0.smithi088.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:40.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T08:58:40.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T08:58:41.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T08:58:41.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T08:58:41.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T08:58:41.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T08:58:41.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T08:58:41.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T08:58:41.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:41.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:41.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T08:58:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T08:58:41.429 INFO:tasks.workunit.client.0.smithi088.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T08:58:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T08:58:41.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T08:58:42.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T08:58:42.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T08:58:42.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T08:58:42.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T08:58:42.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T08:58:42.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T08:58:42.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-04-25T08:58:42.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T08:58:42.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-04-25T08:58:42.833 INFO:tasks.workunit.client.0.smithi088.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-04-25T08:58:42.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 0 = 0 ']' 2024-04-25T08:58:42.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:308: TEST_rep_backfill_unfound: break 2024-04-25T08:58:42.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:314: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T08:58:43.195 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:58:43.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:58:21.016952+0000 0'0 41:71 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:32:31.905105+0000 0 0 2024-04-25T08:58:43.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:58:21.608314+0000 0'0 41:70 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:32:46.525347+0000 0 0 2024-04-25T08:58:43.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:58:22.169564+0000 0'0 41:59 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:30:59.864067+0000 0 0 2024-04-25T08:58:43.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:58:21.507216+0000 0'0 41:70 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:23:19.897578+0000 0 0 2024-04-25T08:58:43.195 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-04-25T08:58:40.000227+0000 30'301 41:896 [2,1,0] 2 [2,1] 2 0'0 2024-04-25T08:57:13.602588+0000 0'0 2024-04-25T08:57:13.602588+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:58:43.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:58:21.016955+0000 0'0 41:71 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:45:49.251163+0000 0 0 2024-04-25T08:58:43.195 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:58:21.608075+0000 0'0 41:56 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:34:06.126040+0000 0 0 2024-04-25T08:58:43.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:58:22.320063+0000 0'0 41:59 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:55:15.989280+0000 0 0 2024-04-25T08:58:43.196 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:58:21.507750+0000 0'0 41:70 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:55:13.401099+0000 0'0 2024-04-25T08:55:13.401099+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:44:24.691241+0000 0 0 2024-04-25T08:58:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T08:58:43.196 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:58:43.196 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T08:58:43.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: grep -q obj250 2024-04-25T08:58:43.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-25T08:58:43.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:318: TEST_rep_backfill_unfound: timeout 5 rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:48.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:319: TEST_rep_backfill_unfound: test 124 = 124 2024-04-25T08:58:48.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:321: TEST_rep_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-25T08:58:48.663 INFO:tasks.workunit.client.0.smithi088.stderr:pg has no unfound objects 2024-04-25T08:58:48.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:323: TEST_rep_backfill_unfound: wait_for_clean 2024-04-25T08:58:48.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:58:48.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:58:48.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:58:48.683 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:58:48.683 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:58:48.683 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:58:48.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:58:48.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:58:48.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:58:48.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:58:48.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:58:48.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:58:48.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:58:48.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:58:48.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:58:49.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:58:49.229 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T08:58:49.229 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T08:58:49.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:58:49.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:49.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:58:49.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953481 2024-04-25T08:58:49.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953481 2024-04-25T08:58:49.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481' 2024-04-25T08:58:49.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:49.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:58:49.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574887 2024-04-25T08:58:49.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574887 2024-04-25T08:58:49.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481 1-55834574887' 2024-04-25T08:58:49.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:58:49.625 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:58:49.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345954 2024-04-25T08:58:49.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345954 2024-04-25T08:58:49.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481 1-55834574887 2-85899345954' 2024-04-25T08:58:49.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:49.784 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-137438953481 2024-04-25T08:58:49.785 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:49.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:58:49.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-137438953481 2024-04-25T08:58:49.786 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:49.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953481 2024-04-25T08:58:49.787 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 137438953481 2024-04-25T08:58:49.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 137438953481' 2024-04-25T08:58:49.787 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:58:50.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953481 -lt 137438953481 2024-04-25T08:58:50.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:50.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574887 2024-04-25T08:58:50.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:50.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:58:50.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574887 2024-04-25T08:58:50.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:50.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574887 2024-04-25T08:58:50.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574887' 2024-04-25T08:58:50.211 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 55834574887 2024-04-25T08:58:50.211 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:58:50.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574887 -lt 55834574887 2024-04-25T08:58:50.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:58:50.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345954 2024-04-25T08:58:50.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:58:50.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:58:50.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345954 2024-04-25T08:58:50.765 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:58:50.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345954 2024-04-25T08:58:50.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345954' 2024-04-25T08:58:50.768 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 85899345954 2024-04-25T08:58:50.768 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345952 -lt 85899345954 2024-04-25T08:58:51.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:58:52.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:58:52.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:58:52.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345955 -lt 85899345954 2024-04-25T08:58:52.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:58:52.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:52.815 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T08:58:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:53.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:53.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:53.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:53.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:53.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:53.360 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:53.797 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:58:53.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:53.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:53.798 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:54.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:58:54.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-04-25T08:58:54.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:58:54.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-04-25T08:58:54.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:54.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:54.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:54.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:54.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:54.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:54.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:54.437 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:54.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:54.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:54.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T08:58:54.827 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:54.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:54.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:55.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T08:58:55.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T08:58:55.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:55.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:55.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:55.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:55.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:55.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:55.381 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:55.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9214 2024-04-25T08:58:55.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9214 '!=' null 2024-04-25T08:58:55.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:55.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:55.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:56.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:56.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:56.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:56.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:56.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:56.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:56.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:56.038 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:56.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T08:58:56.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:56.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:56.430 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:56.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T08:58:56.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:58:56.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:58:56.944 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: seq 1 300 2024-04-25T08:58:56.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:56.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj1 = obj250 ']' 2024-04-25T08:58:56.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj2 = obj250 ']' 2024-04-25T08:58:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj3 = obj250 ']' 2024-04-25T08:58:57.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj4 = obj250 ']' 2024-04-25T08:58:57.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj5 = obj250 ']' 2024-04-25T08:58:57.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj6 = obj250 ']' 2024-04-25T08:58:57.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj7 = obj250 ']' 2024-04-25T08:58:57.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj8 = obj250 ']' 2024-04-25T08:58:57.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj9 = obj250 ']' 2024-04-25T08:58:57.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj10 = obj250 ']' 2024-04-25T08:58:57.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj11 = obj250 ']' 2024-04-25T08:58:57.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj12 = obj250 ']' 2024-04-25T08:58:57.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj13 = obj250 ']' 2024-04-25T08:58:57.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj14 = obj250 ']' 2024-04-25T08:58:57.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj15 = obj250 ']' 2024-04-25T08:58:57.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj16 = obj250 ']' 2024-04-25T08:58:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj17 = obj250 ']' 2024-04-25T08:58:57.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj18 = obj250 ']' 2024-04-25T08:58:57.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj19 = obj250 ']' 2024-04-25T08:58:57.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj20 = obj250 ']' 2024-04-25T08:58:57.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj21 = obj250 ']' 2024-04-25T08:58:57.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj22 = obj250 ']' 2024-04-25T08:58:57.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.768 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj23 = obj250 ']' 2024-04-25T08:58:57.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj24 = obj250 ']' 2024-04-25T08:58:57.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj25 = obj250 ']' 2024-04-25T08:58:57.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj26 = obj250 ']' 2024-04-25T08:58:57.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj27 = obj250 ']' 2024-04-25T08:58:57.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj28 = obj250 ']' 2024-04-25T08:58:57.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:57.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj29 = obj250 ']' 2024-04-25T08:58:57.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:57.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj30 = obj250 ']' 2024-04-25T08:58:58.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj31 = obj250 ']' 2024-04-25T08:58:58.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj32 = obj250 ']' 2024-04-25T08:58:58.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj33 = obj250 ']' 2024-04-25T08:58:58.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj34 = obj250 ']' 2024-04-25T08:58:58.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj35 = obj250 ']' 2024-04-25T08:58:58.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj36 = obj250 ']' 2024-04-25T08:58:58.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj37 = obj250 ']' 2024-04-25T08:58:58.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj38 = obj250 ']' 2024-04-25T08:58:58.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj39 = obj250 ']' 2024-04-25T08:58:58.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj40 = obj250 ']' 2024-04-25T08:58:58.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj41 = obj250 ']' 2024-04-25T08:58:58.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj42 = obj250 ']' 2024-04-25T08:58:58.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj43 = obj250 ']' 2024-04-25T08:58:58.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj44 = obj250 ']' 2024-04-25T08:58:58.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj45 = obj250 ']' 2024-04-25T08:58:58.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj46 = obj250 ']' 2024-04-25T08:58:58.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj47 = obj250 ']' 2024-04-25T08:58:58.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj48 = obj250 ']' 2024-04-25T08:58:58.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj49 = obj250 ']' 2024-04-25T08:58:58.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj50 = obj250 ']' 2024-04-25T08:58:58.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj51 = obj250 ']' 2024-04-25T08:58:58.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj52 = obj250 ']' 2024-04-25T08:58:58.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj53 = obj250 ']' 2024-04-25T08:58:58.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj54 = obj250 ']' 2024-04-25T08:58:58.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj55 = obj250 ']' 2024-04-25T08:58:58.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj56 = obj250 ']' 2024-04-25T08:58:58.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj57 = obj250 ']' 2024-04-25T08:58:58.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj58 = obj250 ']' 2024-04-25T08:58:58.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj59 = obj250 ']' 2024-04-25T08:58:58.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj60 = obj250 ']' 2024-04-25T08:58:58.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj61 = obj250 ']' 2024-04-25T08:58:58.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:58.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:58.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj62 = obj250 ']' 2024-04-25T08:58:58.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj63 = obj250 ']' 2024-04-25T08:58:59.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj64 = obj250 ']' 2024-04-25T08:58:59.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj65 = obj250 ']' 2024-04-25T08:58:59.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj66 = obj250 ']' 2024-04-25T08:58:59.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj67 = obj250 ']' 2024-04-25T08:58:59.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj68 = obj250 ']' 2024-04-25T08:58:59.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj69 = obj250 ']' 2024-04-25T08:58:59.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj70 = obj250 ']' 2024-04-25T08:58:59.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj71 = obj250 ']' 2024-04-25T08:58:59.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj72 = obj250 ']' 2024-04-25T08:58:59.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj73 = obj250 ']' 2024-04-25T08:58:59.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj74 = obj250 ']' 2024-04-25T08:58:59.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj75 = obj250 ']' 2024-04-25T08:58:59.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj76 = obj250 ']' 2024-04-25T08:58:59.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj77 = obj250 ']' 2024-04-25T08:58:59.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj78 = obj250 ']' 2024-04-25T08:58:59.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj79 = obj250 ']' 2024-04-25T08:58:59.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj80 = obj250 ']' 2024-04-25T08:58:59.518 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj81 = obj250 ']' 2024-04-25T08:58:59.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj82 = obj250 ']' 2024-04-25T08:58:59.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj83 = obj250 ']' 2024-04-25T08:58:59.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj84 = obj250 ']' 2024-04-25T08:58:59.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj85 = obj250 ']' 2024-04-25T08:58:59.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj86 = obj250 ']' 2024-04-25T08:58:59.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj87 = obj250 ']' 2024-04-25T08:58:59.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj88 = obj250 ']' 2024-04-25T08:58:59.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj89 = obj250 ']' 2024-04-25T08:58:59.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj90 = obj250 ']' 2024-04-25T08:58:59.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj91 = obj250 ']' 2024-04-25T08:58:59.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj92 = obj250 ']' 2024-04-25T08:58:59.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj93 = obj250 ']' 2024-04-25T08:58:59.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj94 = obj250 ']' 2024-04-25T08:58:59.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:58:59.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:58:59.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj95 = obj250 ']' 2024-04-25T08:58:59.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj96 = obj250 ']' 2024-04-25T08:59:00.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj97 = obj250 ']' 2024-04-25T08:59:00.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj98 = obj250 ']' 2024-04-25T08:59:00.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj99 = obj250 ']' 2024-04-25T08:59:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj100 = obj250 ']' 2024-04-25T08:59:00.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj101 = obj250 ']' 2024-04-25T08:59:00.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj101 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj102 = obj250 ']' 2024-04-25T08:59:00.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj102 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj103 = obj250 ']' 2024-04-25T08:59:00.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj103 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj104 = obj250 ']' 2024-04-25T08:59:00.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj104 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj105 = obj250 ']' 2024-04-25T08:59:00.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj105 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj106 = obj250 ']' 2024-04-25T08:59:00.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj106 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj107 = obj250 ']' 2024-04-25T08:59:00.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj107 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj108 = obj250 ']' 2024-04-25T08:59:00.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj108 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj109 = obj250 ']' 2024-04-25T08:59:00.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj109 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj110 = obj250 ']' 2024-04-25T08:59:00.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj110 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj111 = obj250 ']' 2024-04-25T08:59:00.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj111 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj112 = obj250 ']' 2024-04-25T08:59:00.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj112 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj113 = obj250 ']' 2024-04-25T08:59:00.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj113 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj114 = obj250 ']' 2024-04-25T08:59:00.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj114 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj115 = obj250 ']' 2024-04-25T08:59:00.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj115 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj116 = obj250 ']' 2024-04-25T08:59:00.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj116 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj117 = obj250 ']' 2024-04-25T08:59:00.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj117 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj118 = obj250 ']' 2024-04-25T08:59:00.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj118 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj119 = obj250 ']' 2024-04-25T08:59:00.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj119 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj120 = obj250 ']' 2024-04-25T08:59:00.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj120 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj121 = obj250 ']' 2024-04-25T08:59:00.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj121 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj122 = obj250 ']' 2024-04-25T08:59:00.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj122 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj123 = obj250 ']' 2024-04-25T08:59:00.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj123 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj124 = obj250 ']' 2024-04-25T08:59:00.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj124 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj125 = obj250 ']' 2024-04-25T08:59:00.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj125 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:00.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:00.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj126 = obj250 ']' 2024-04-25T08:59:00.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj126 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj127 = obj250 ']' 2024-04-25T08:59:01.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj127 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj128 = obj250 ']' 2024-04-25T08:59:01.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj128 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj129 = obj250 ']' 2024-04-25T08:59:01.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj129 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj130 = obj250 ']' 2024-04-25T08:59:01.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj130 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj131 = obj250 ']' 2024-04-25T08:59:01.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj131 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj132 = obj250 ']' 2024-04-25T08:59:01.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj132 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj133 = obj250 ']' 2024-04-25T08:59:01.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj133 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj134 = obj250 ']' 2024-04-25T08:59:01.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj134 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj135 = obj250 ']' 2024-04-25T08:59:01.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj135 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj136 = obj250 ']' 2024-04-25T08:59:01.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj136 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj137 = obj250 ']' 2024-04-25T08:59:01.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj137 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj138 = obj250 ']' 2024-04-25T08:59:01.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj138 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj139 = obj250 ']' 2024-04-25T08:59:01.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj139 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj140 = obj250 ']' 2024-04-25T08:59:01.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj140 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj141 = obj250 ']' 2024-04-25T08:59:01.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj141 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj142 = obj250 ']' 2024-04-25T08:59:01.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj142 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj143 = obj250 ']' 2024-04-25T08:59:01.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj143 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj144 = obj250 ']' 2024-04-25T08:59:01.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj144 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj145 = obj250 ']' 2024-04-25T08:59:01.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj145 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj146 = obj250 ']' 2024-04-25T08:59:01.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj146 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj147 = obj250 ']' 2024-04-25T08:59:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj147 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj148 = obj250 ']' 2024-04-25T08:59:01.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj148 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj149 = obj250 ']' 2024-04-25T08:59:01.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj149 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj150 = obj250 ']' 2024-04-25T08:59:01.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj150 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj151 = obj250 ']' 2024-04-25T08:59:01.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj151 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj152 = obj250 ']' 2024-04-25T08:59:01.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj152 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj153 = obj250 ']' 2024-04-25T08:59:01.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj153 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj154 = obj250 ']' 2024-04-25T08:59:01.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj154 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj155 = obj250 ']' 2024-04-25T08:59:01.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj155 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj156 = obj250 ']' 2024-04-25T08:59:01.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj156 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj157 = obj250 ']' 2024-04-25T08:59:01.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj157 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj158 = obj250 ']' 2024-04-25T08:59:01.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj158 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj159 = obj250 ']' 2024-04-25T08:59:02.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj159 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj160 = obj250 ']' 2024-04-25T08:59:02.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj160 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj161 = obj250 ']' 2024-04-25T08:59:02.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj161 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj162 = obj250 ']' 2024-04-25T08:59:02.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj162 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj163 = obj250 ']' 2024-04-25T08:59:02.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj163 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj164 = obj250 ']' 2024-04-25T08:59:02.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj164 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj165 = obj250 ']' 2024-04-25T08:59:02.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj165 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj166 = obj250 ']' 2024-04-25T08:59:02.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj166 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj167 = obj250 ']' 2024-04-25T08:59:02.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj167 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj168 = obj250 ']' 2024-04-25T08:59:02.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj168 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj169 = obj250 ']' 2024-04-25T08:59:02.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj169 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj170 = obj250 ']' 2024-04-25T08:59:02.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj170 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj171 = obj250 ']' 2024-04-25T08:59:02.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj171 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj172 = obj250 ']' 2024-04-25T08:59:02.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj172 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj173 = obj250 ']' 2024-04-25T08:59:02.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj173 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj174 = obj250 ']' 2024-04-25T08:59:02.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj174 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj175 = obj250 ']' 2024-04-25T08:59:02.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj175 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj176 = obj250 ']' 2024-04-25T08:59:02.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj176 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj177 = obj250 ']' 2024-04-25T08:59:02.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj177 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj178 = obj250 ']' 2024-04-25T08:59:02.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj178 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj179 = obj250 ']' 2024-04-25T08:59:02.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj179 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj180 = obj250 ']' 2024-04-25T08:59:02.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj180 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj181 = obj250 ']' 2024-04-25T08:59:02.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj181 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj182 = obj250 ']' 2024-04-25T08:59:02.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj182 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj183 = obj250 ']' 2024-04-25T08:59:02.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj183 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj184 = obj250 ']' 2024-04-25T08:59:02.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj184 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj185 = obj250 ']' 2024-04-25T08:59:02.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj185 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj186 = obj250 ']' 2024-04-25T08:59:02.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj186 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.900 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj187 = obj250 ']' 2024-04-25T08:59:02.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj187 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj188 = obj250 ']' 2024-04-25T08:59:02.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj188 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj189 = obj250 ']' 2024-04-25T08:59:02.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj189 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:02.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:02.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj190 = obj250 ']' 2024-04-25T08:59:02.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj190 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj191 = obj250 ']' 2024-04-25T08:59:03.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj191 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj192 = obj250 ']' 2024-04-25T08:59:03.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj192 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj193 = obj250 ']' 2024-04-25T08:59:03.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj193 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj194 = obj250 ']' 2024-04-25T08:59:03.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj194 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj195 = obj250 ']' 2024-04-25T08:59:03.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj195 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj196 = obj250 ']' 2024-04-25T08:59:03.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj196 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj197 = obj250 ']' 2024-04-25T08:59:03.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj197 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj198 = obj250 ']' 2024-04-25T08:59:03.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj198 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj199 = obj250 ']' 2024-04-25T08:59:03.257 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj199 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj200 = obj250 ']' 2024-04-25T08:59:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj200 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj201 = obj250 ']' 2024-04-25T08:59:03.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj201 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj202 = obj250 ']' 2024-04-25T08:59:03.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj202 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj203 = obj250 ']' 2024-04-25T08:59:03.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj203 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj204 = obj250 ']' 2024-04-25T08:59:03.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj204 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj205 = obj250 ']' 2024-04-25T08:59:03.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj205 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj206 = obj250 ']' 2024-04-25T08:59:03.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj206 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj207 = obj250 ']' 2024-04-25T08:59:03.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj207 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj208 = obj250 ']' 2024-04-25T08:59:03.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj208 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj209 = obj250 ']' 2024-04-25T08:59:03.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj209 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj210 = obj250 ']' 2024-04-25T08:59:03.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj210 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj211 = obj250 ']' 2024-04-25T08:59:03.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj211 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj212 = obj250 ']' 2024-04-25T08:59:03.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj212 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj213 = obj250 ']' 2024-04-25T08:59:03.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj213 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj214 = obj250 ']' 2024-04-25T08:59:03.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj214 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj215 = obj250 ']' 2024-04-25T08:59:03.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj215 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj216 = obj250 ']' 2024-04-25T08:59:03.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj216 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj217 = obj250 ']' 2024-04-25T08:59:03.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj217 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.850 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj218 = obj250 ']' 2024-04-25T08:59:03.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj218 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj219 = obj250 ']' 2024-04-25T08:59:03.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj219 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj220 = obj250 ']' 2024-04-25T08:59:03.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj220 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj221 = obj250 ']' 2024-04-25T08:59:03.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj221 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj222 = obj250 ']' 2024-04-25T08:59:03.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj222 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:03.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:03.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj223 = obj250 ']' 2024-04-25T08:59:03.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj223 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj224 = obj250 ']' 2024-04-25T08:59:04.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj224 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj225 = obj250 ']' 2024-04-25T08:59:04.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj225 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj226 = obj250 ']' 2024-04-25T08:59:04.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj226 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj227 = obj250 ']' 2024-04-25T08:59:04.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj227 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj228 = obj250 ']' 2024-04-25T08:59:04.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj228 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj229 = obj250 ']' 2024-04-25T08:59:04.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj229 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj230 = obj250 ']' 2024-04-25T08:59:04.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj230 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj231 = obj250 ']' 2024-04-25T08:59:04.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj231 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj232 = obj250 ']' 2024-04-25T08:59:04.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj232 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj233 = obj250 ']' 2024-04-25T08:59:04.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj233 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj234 = obj250 ']' 2024-04-25T08:59:04.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj234 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj235 = obj250 ']' 2024-04-25T08:59:04.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj235 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj236 = obj250 ']' 2024-04-25T08:59:04.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj236 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj237 = obj250 ']' 2024-04-25T08:59:04.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj237 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj238 = obj250 ']' 2024-04-25T08:59:04.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj238 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj239 = obj250 ']' 2024-04-25T08:59:04.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj239 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj240 = obj250 ']' 2024-04-25T08:59:04.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj240 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj241 = obj250 ']' 2024-04-25T08:59:04.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj241 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj242 = obj250 ']' 2024-04-25T08:59:04.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj242 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj243 = obj250 ']' 2024-04-25T08:59:04.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj243 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj244 = obj250 ']' 2024-04-25T08:59:04.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj244 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj245 = obj250 ']' 2024-04-25T08:59:04.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj245 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj246 = obj250 ']' 2024-04-25T08:59:04.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj246 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj247 = obj250 ']' 2024-04-25T08:59:04.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj247 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj248 = obj250 ']' 2024-04-25T08:59:04.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj248 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj249 = obj250 ']' 2024-04-25T08:59:04.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj249 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj250 = obj250 ']' 2024-04-25T08:59:04.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:329: TEST_rep_backfill_unfound: rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.908 INFO:tasks.workunit.client.0.smithi088.stderr:error getting test-pool/obj250: (2) No such file or directory 2024-04-25T08:59:04.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj251 = obj250 ']' 2024-04-25T08:59:04.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj251 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj252 = obj250 ']' 2024-04-25T08:59:04.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj252 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:04.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj253 = obj250 ']' 2024-04-25T08:59:04.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj253 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:04.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj254 = obj250 ']' 2024-04-25T08:59:05.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj254 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj255 = obj250 ']' 2024-04-25T08:59:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj255 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj256 = obj250 ']' 2024-04-25T08:59:05.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj256 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj257 = obj250 ']' 2024-04-25T08:59:05.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj257 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj258 = obj250 ']' 2024-04-25T08:59:05.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj258 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj259 = obj250 ']' 2024-04-25T08:59:05.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj259 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj260 = obj250 ']' 2024-04-25T08:59:05.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj260 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj261 = obj250 ']' 2024-04-25T08:59:05.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj261 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj262 = obj250 ']' 2024-04-25T08:59:05.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj262 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj263 = obj250 ']' 2024-04-25T08:59:05.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj263 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj264 = obj250 ']' 2024-04-25T08:59:05.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj264 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj265 = obj250 ']' 2024-04-25T08:59:05.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj265 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj266 = obj250 ']' 2024-04-25T08:59:05.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj266 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj267 = obj250 ']' 2024-04-25T08:59:05.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj267 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj268 = obj250 ']' 2024-04-25T08:59:05.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj268 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj269 = obj250 ']' 2024-04-25T08:59:05.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj269 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj270 = obj250 ']' 2024-04-25T08:59:05.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj270 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj271 = obj250 ']' 2024-04-25T08:59:05.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj271 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.533 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj272 = obj250 ']' 2024-04-25T08:59:05.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj272 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj273 = obj250 ']' 2024-04-25T08:59:05.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj273 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj274 = obj250 ']' 2024-04-25T08:59:05.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj274 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj275 = obj250 ']' 2024-04-25T08:59:05.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj275 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.652 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj276 = obj250 ']' 2024-04-25T08:59:05.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj276 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj277 = obj250 ']' 2024-04-25T08:59:05.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj277 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj278 = obj250 ']' 2024-04-25T08:59:05.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj278 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj279 = obj250 ']' 2024-04-25T08:59:05.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj279 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.771 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj280 = obj250 ']' 2024-04-25T08:59:05.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj280 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj281 = obj250 ']' 2024-04-25T08:59:05.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj281 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj282 = obj250 ']' 2024-04-25T08:59:05.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj282 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj283 = obj250 ']' 2024-04-25T08:59:05.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj283 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj284 = obj250 ']' 2024-04-25T08:59:05.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj284 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj285 = obj250 ']' 2024-04-25T08:59:05.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj285 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:05.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:05.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj286 = obj250 ']' 2024-04-25T08:59:05.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj286 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj287 = obj250 ']' 2024-04-25T08:59:06.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj287 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj288 = obj250 ']' 2024-04-25T08:59:06.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj288 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj289 = obj250 ']' 2024-04-25T08:59:06.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj289 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj290 = obj250 ']' 2024-04-25T08:59:06.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj290 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj291 = obj250 ']' 2024-04-25T08:59:06.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj291 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj292 = obj250 ']' 2024-04-25T08:59:06.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj292 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj293 = obj250 ']' 2024-04-25T08:59:06.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj293 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj294 = obj250 ']' 2024-04-25T08:59:06.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj294 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj295 = obj250 ']' 2024-04-25T08:59:06.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj295 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj296 = obj250 ']' 2024-04-25T08:59:06.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj296 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj297 = obj250 ']' 2024-04-25T08:59:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj297 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj298 = obj250 ']' 2024-04-25T08:59:06.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj298 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj299 = obj250 ']' 2024-04-25T08:59:06.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj299 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T08:59:06.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj300 = obj250 ']' 2024-04-25T08:59:06.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj300 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:336: TEST_rep_backfill_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T08:59:06.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:338: TEST_rep_backfill_unfound: delete_pool test-pool 2024-04-25T08:59:06.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-04-25T08:59:06.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-04-25T08:59:06.942 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test-pool' does not exist 2024-04-25T08:59:06.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T08:59:06.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:59:06.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:59:06.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T08:59:06.959 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:59:06.960 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:59:06.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:59:06.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:59:06.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:59:07.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:59:07.271 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:59:07.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:59:07.272 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:59:07.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:59:07.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:59:07.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:59:07.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:59:07.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:59:07.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:59:07.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:59:07.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:59:07.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:59:07.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T08:59:07.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:59:07.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:07.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:07.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:59:07.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:59:07.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:59:07.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:59:07.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:59:07.328 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:59:07.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:59:07.329 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:59:07.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:59:07.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:59:07.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:59:07.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:59:07.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:59:07.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:59:07.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:59:07.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:59:07.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:59:07.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T08:59:07.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:59:07.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:07.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:07.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T08:59:07.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:59:07.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:59:07.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T08:59:07.338 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:59:07.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:07.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:07.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.389694 2024-04-25T08:59:07.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T08:59:07.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T08:59:07.820 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:59:07.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:59:07.821 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:59:07.821 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:59:07.821 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:07.822 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:07.822 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:59:07.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T08:59:08.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:59:08.067 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:59:08.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:59:08.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:59:08.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:59:08.068 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:59:08.068 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:59:08.068 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:59:08.068 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:59:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:59:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:08.069 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-mon.a.asok 2024-04-25T08:59:08.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:59:08.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.389694/ceph-mon.a.asok config get fsid 2024-04-25T08:59:08.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:59:08.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:59:08.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:59:08.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:59:08.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:59:08.186 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:59:08.186 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:59:08.187 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:59:08.187 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:59:08.187 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:08.187 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:08.187 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-mon.a.asok 2024-04-25T08:59:08.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:59:08.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.389694/ceph-mon.a.asok config get mon_host 2024-04-25T08:59:08.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T08:59:08.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:59:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:59:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:59:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:59:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T08:59:08.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:59:09.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:59:09.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:59:09.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:59:09.208 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:59:09.208 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:09.209 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:09.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:59:09.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:59:09.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:59:09.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T08:59:09.586 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_recovery_unfound td/osd-rep-recov-eio.sh 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:343: TEST_rep_recovery_unfound: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:344: TEST_rep_recovery_unfound: local objname=myobject 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:345: TEST_rep_recovery_unfound: local lastobj=100 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:347: TEST_rep_recovery_unfound: local testobj=obj75 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:349: TEST_rep_recovery_unfound: setup_osds 3 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T08:59:09.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T08:59:09.605 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-04-25T08:59:09.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-04-25T08:59:09.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:59:09.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:59:09.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:59:09.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44a04b79-a56e-43a6-bdf7-57e9eb6f7d03 2024-04-25T08:59:09.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 44a04b79-a56e-43a6-bdf7-57e9eb6f7d03' 2024-04-25T08:59:09.616 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 44a04b79-a56e-43a6-bdf7-57e9eb6f7d03 2024-04-25T08:59:09.616 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:59:09.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdGypmfNulJhAACB50mpcrp0tBAbvMN6LC0w== 2024-04-25T08:59:09.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdGypmfNulJhAACB50mpcrp0tBAbvMN6LC0w=="}' 2024-04-25T08:59:09.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44a04b79-a56e-43a6-bdf7-57e9eb6f7d03 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:59:09.920 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:59:09.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T08:59:09.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdGypmfNulJhAACB50mpcrp0tBAbvMN6LC0w== --osd-uuid 44a04b79-a56e-43a6-bdf7-57e9eb6f7d03 2024-04-25T08:59:09.966 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:09.977+0000 7f8b3c44d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:09.980 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:09.993+0000 7f8b3c44d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:09.981 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:09.993+0000 7f8b3c44d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:09.982 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:09.993+0000 7f8b3c44d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:59:09.982 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:09.993+0000 7f8b3c44d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:59:09.982 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:09.993+0000 7f8b3c44d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T08:59:10.020 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:10.033+0000 7f8b3c44d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T08:59:16.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T08:59:16.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:59:16.061 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T08:59:16.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:59:16.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:59:16.638 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T08:59:16.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:59:16.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:59:16.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:59:16.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:59:16.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:59:16.666 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:16.677+0000 7f9cd88ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:16.668 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:16.681+0000 7f9cd88ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:16.670 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:16.681+0000 7f9cd88ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:17.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:59:17.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:59:17.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:59:17.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:59:17.077 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:59:17.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:59:17.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:17.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:59:17.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:17.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:17.225 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:17.237+0000 7f9cd88ce7c0 -1 Falling back to public interface 2024-04-25T08:59:17.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:18.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:18.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:18.472 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:59:18.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:59:18.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:18.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:18.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:19.832 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:19.841+0000 7f9cd88ce7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:59:19.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:19.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:19.878 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:59:19.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:59:19.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:19.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:20.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:21.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:21.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:21.542 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:59:21.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:59:21.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:21.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:22.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:23.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:23.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:23.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:59:23.011 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:59:23.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:23.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:23.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:24.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:24.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:24.449 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T08:59:24.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:59:24.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:24.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:24.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:25.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:25.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:25.968 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T08:59:25.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:59:25.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:25.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:26.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:27.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:27.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:27.391 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T08:59:27.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:59:27.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:27.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:27.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:28.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:28.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:28.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:59:28.834 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T08:59:28.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:28.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:29.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:30.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:30.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:30.331 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T08:59:30.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:59:30.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:30.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:30.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:31.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:31.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:31.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:59:31.829 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T08:59:31.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:31.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:32.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:33.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:33.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:33.245 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T08:59:33.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:59:33.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:33.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:33.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:34.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:34.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:34.744 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T08:59:34.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:59:34.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:34.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:35.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:36.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:36.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:36.459 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T08:59:36.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:59:36.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:36.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:37.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:38.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:38.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:38.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:59:38.470 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T08:59:38.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:38.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:39.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:40.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:40.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:40.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:59:40.418 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T08:59:40.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:40.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:41.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:42.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:42.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:42.004 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T08:59:42.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:59:42.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:42.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:43.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:44.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:44.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:44.229 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T08:59:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:59:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:44.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:44.663 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1771675377,v1:127.0.0.1:6803/1771675377] [v2:127.0.0.1:6804/1771675377,v1:127.0.0.1:6805/1771675377] exists,up 44a04b79-a56e-43a6-bdf7-57e9eb6f7d03 2024-04-25T08:59:44.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:59:44.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:59:44.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:59:44.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T08:59:44.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:59:44.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T08:59:44.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:59:44.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:59:44.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:59:44.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:59:44.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:44.668 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T08:59:44.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:59:44.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:59:44.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:59:44.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T08:59:44.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:59:44.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f440e050-df9a-4048-a019-65c831dc8b04 2024-04-25T08:59:44.671 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 f440e050-df9a-4048-a019-65c831dc8b04 2024-04-25T08:59:44.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f440e050-df9a-4048-a019-65c831dc8b04' 2024-04-25T08:59:44.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:59:44.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAGypmkAoZKhAA1hNe9gba+PE4RALBE6yz/Q== 2024-04-25T08:59:44.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAGypmkAoZKhAA1hNe9gba+PE4RALBE6yz/Q=="}' 2024-04-25T08:59:44.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f440e050-df9a-4048-a019-65c831dc8b04 -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:59:45.328 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:59:45.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T08:59:45.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAGypmkAoZKhAA1hNe9gba+PE4RALBE6yz/Q== --osd-uuid f440e050-df9a-4048-a019-65c831dc8b04 2024-04-25T08:59:45.385 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:45.397+0000 7fec9cdf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:45.403 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:45.413+0000 7fec9cdf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:45.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:45.417+0000 7fec9cdf57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:45.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:45.417+0000 7fec9cdf57c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:59:45.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:45.417+0000 7fec9cdf57c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:59:45.405 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:45.417+0000 7fec9cdf57c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T08:59:45.449 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:45.461+0000 7fec9cdf57c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T08:59:52.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T08:59:52.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:59:52.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:59:52.168 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T08:59:52.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:59:52.713 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T08:59:52.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:59:52.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:59:52.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:59:52.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:59:52.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:59:52.747 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:52.756+0000 7f4fbdd3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:52.775 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:52.784+0000 7f4fbdd3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:52.776 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:52.788+0000 7f4fbdd3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:53.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:59:53.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:59:53.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:59:53.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:59:53.132 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T08:59:53.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:59:53.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:53.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:59:53.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:53.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:53.330 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:53.340+0000 7f4fbdd3a7c0 -1 Falling back to public interface 2024-04-25T08:59:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:54.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:54.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:54.561 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T08:59:54.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:59:54.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:54.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:54.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:55.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:55.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:55.992 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T08:59:55.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:59:55.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:55.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:56.037 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T08:59:56.048+0000 7f4fbdd3a7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:59:56.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:57.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:57.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:57.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:59:57.576 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T08:59:57.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:57.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:58.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:59.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:59.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:59.058 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T08:59:59.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:59:59.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:59.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:59.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:00.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:00.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:00.530 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:00:00.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:00:00.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:00.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:01.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:02.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:02.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:02.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:00:02.010 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:00:02.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:02.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:02.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:03.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:03.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:03.615 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:00:03.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:00:03.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:03.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:04.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:05.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:05.272 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:00:05.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:00:05.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:05.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:05.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:06.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:06.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:06.696 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:00:06.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:00:06.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:06.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:07.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:08.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:08.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:08.178 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:00:08.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:00:08.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:08.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:08.798 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:09.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:09.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:09.800 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:09.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:10.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:11.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:11.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:11.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:00:11.523 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:00:11.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:11.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:12.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:13.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:13.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:13.664 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:00:13.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:00:13.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:13.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:14.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:15.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:15.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:15.884 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:00:15.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:00:15.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:15.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:17.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:18.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:18.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:18.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:00:18.148 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:00:18.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:18.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:19.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:20.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:20.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:20.193 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:00:20.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:00:20.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:20.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:20.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:21.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:21.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:21.763 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:00:21.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:00:21.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:21.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:22.413 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/593374178,v1:127.0.0.1:6811/593374178] [v2:127.0.0.1:6812/593374178,v1:127.0.0.1:6813/593374178] exists,up f440e050-df9a-4048-a019-65c831dc8b04 2024-04-25T09:00:22.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:00:22.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:00:22.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:00:22.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:00:22.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:00:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:00:22.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:00:22.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:00:22.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:00:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:00:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:22.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T09:00:22.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T09:00:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T09:00:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:00:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5abec9f2-5bcb-4713-be6f-32df4553d46c 2024-04-25T09:00:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5abec9f2-5bcb-4713-be6f-32df4553d46c' 2024-04-25T09:00:22.424 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 5abec9f2-5bcb-4713-be6f-32df4553d46c 2024-04-25T09:00:22.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:00:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmGypmIIAlGxAANflTLuH5Jww0etnX+4x3WA== 2024-04-25T09:00:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmGypmIIAlGxAANflTLuH5Jww0etnX+4x3WA=="}' 2024-04-25T09:00:22.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5abec9f2-5bcb-4713-be6f-32df4553d46c -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:00:22.897 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:00:22.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:00:22.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmGypmIIAlGxAANflTLuH5Jww0etnX+4x3WA== --osd-uuid 5abec9f2-5bcb-4713-be6f-32df4553d46c 2024-04-25T09:00:22.950 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:22.960+0000 7fe36a87e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:22.955 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:22.964+0000 7fe36a87e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:22.956 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:22.968+0000 7fe36a87e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:22.957 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:22.968+0000 7fe36a87e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:00:22.957 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:22.968+0000 7fe36a87e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:00:22.957 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:22.968+0000 7fe36a87e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:00:22.987 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:22.996+0000 7fe36a87e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:00:29.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:00:29.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:00:29.238 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T09:00:29.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:00:29.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:00:29.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:00:29.791 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:00:29.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:00:29.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:00:29.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:00:29.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:00:29.824 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:29.832+0000 7f90e84ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:29.826 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:29.836+0000 7f90e84ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:29.827 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:29.836+0000 7f90e84ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:30.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:00:30.219 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:00:30.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:00:30.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:00:30.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:00:30.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:30.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:00:30.220 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:00:30.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:30.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:30.417 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:30.428+0000 7f90e84ff7c0 -1 Falling back to public interface 2024-04-25T09:00:30.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:31.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:31.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:31.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:00:31.672 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:00:31.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:31.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:32.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:32.943 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:00:32.952+0000 7f90e84ff7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:00:33.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:33.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:33.088 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:00:33.088 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:00:33.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:33.089 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:33.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:34.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:34.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:34.691 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:00:34.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:00:34.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:34.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:35.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:36.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:36.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:36.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:00:36.205 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:00:36.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:36.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:36.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:37.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:37.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:37.746 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:00:37.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:00:37.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:37.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:38.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:39.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:39.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:00:39.288 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:00:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:39.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:39.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:40.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:40.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:40.811 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:00:40.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:00:40.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:40.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:41.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:42.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:42.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:42.325 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:00:42.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:00:42.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:42.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:42.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:43.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:43.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:43.753 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:00:43.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:00:43.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:43.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:44.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:45.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:45.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:45.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:00:45.235 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:00:45.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:45.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:45.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:46.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:46.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:46.805 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:00:46.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:00:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:46.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:47.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:48.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:48.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:48.935 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:00:48.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:00:48.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:48.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:49.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:50.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:50.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:50.995 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:00:50.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:00:50.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:50.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:51.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:52.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:52.979 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:00:52.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:52.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:00:52.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:52.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:54.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:55.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:55.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:55.052 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:00:55.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:00:55.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:55.053 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:56.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:57.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:57.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:57.094 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:00:57.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:00:57.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:57.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:58.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:59.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:59.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:59.032 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:00:59.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:00:59.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:59.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:59.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:00.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:00.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:00.481 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:01:00.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:01:00.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:00.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:01:00.919 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1036271552,v1:127.0.0.1:6819/1036271552] [v2:127.0.0.1:6820/1036271552,v1:127.0.0.1:6821/1036271552] exists,up 5abec9f2-5bcb-4713-be6f-32df4553d46c 2024-04-25T09:01:00.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:01:00.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:01:00.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:01:00.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:01:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:01:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:01:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:01:00.921 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:01:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:01:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:01:00.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:01:00.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:01:00.922 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:01:01.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:01:01.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:01:01.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:01:01.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:01:01.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:01:01.024 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:01:01.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:01:01.426 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:01:01.426 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:01:01.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:01:01.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:01.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:01:01.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803798 2024-04-25T09:01:01.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803798 2024-04-25T09:01:01.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798' 2024-04-25T09:01:01.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:01.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:01:01.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607567 2024-04-25T09:01:01.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607567 2024-04-25T09:01:01.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798 1-51539607567' 2024-04-25T09:01:01.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:01.763 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:01:01.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378631 2024-04-25T09:01:01.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378631 2024-04-25T09:01:01.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798 1-51539607567 2-81604378631' 2024-04-25T09:01:01.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:01.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803798 2024-04-25T09:01:01.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:01.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:01:01.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803798 2024-04-25T09:01:01.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:01.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803798 2024-04-25T09:01:01.919 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803798 2024-04-25T09:01:01.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2024-04-25T09:01:01.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:01:02.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803798 -lt 25769803798 2024-04-25T09:01:02.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:02.316 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607567 2024-04-25T09:01:02.317 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:02.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:01:02.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607567 2024-04-25T09:01:02.319 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:02.321 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607567 2024-04-25T09:01:02.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607567 2024-04-25T09:01:02.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607567' 2024-04-25T09:01:02.321 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:01:02.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607567 -lt 51539607567 2024-04-25T09:01:02.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:02.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378631 2024-04-25T09:01:02.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:02.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:01:02.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378631 2024-04-25T09:01:02.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:02.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378631 2024-04-25T09:01:02.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378631' 2024-04-25T09:01:02.845 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 81604378631 2024-04-25T09:01:02.845 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:01:03.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378629 -lt 81604378631 2024-04-25T09:01:03.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:01:04.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:01:04.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:01:04.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378631 -lt 81604378631 2024-04-25T09:01:04.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:01:04.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:04.738 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:05.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:01:05.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:05.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:05.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:05.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:05.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:05.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:05.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:05.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:01:05.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:05.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:05.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:01:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:01:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:01:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:351: TEST_rep_recovery_unfound: local poolname=test-pool 2024-04-25T09:01:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:352: TEST_rep_recovery_unfound: create_pool test-pool 1 1 2024-04-25T09:01:06.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-04-25T09:01:06.808 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test-pool' already exists 2024-04-25T09:01:06.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:01:07.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:353: TEST_rep_recovery_unfound: wait_for_clean 2024-04-25T09:01:07.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:01:07.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:01:07.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:01:07.826 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:01:07.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:01:07.827 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:01:07.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:01:07.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:01:07.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:01:07.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:01:07.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:01:07.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:01:07.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:01:07.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:01:07.953 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:01:08.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:01:08.569 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:01:08.570 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:01:08.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:01:08.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:08.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:01:08.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803801 2024-04-25T09:01:08.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803801 2024-04-25T09:01:08.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801' 2024-04-25T09:01:08.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:08.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:01:08.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607570 2024-04-25T09:01:08.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607570 2024-04-25T09:01:08.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801 1-51539607570' 2024-04-25T09:01:08.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:08.893 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:01:09.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378634 2024-04-25T09:01:09.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378634 2024-04-25T09:01:09.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801 1-51539607570 2-81604378634' 2024-04-25T09:01:09.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803801 2024-04-25T09:01:09.047 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:09.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:01:09.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803801 2024-04-25T09:01:09.048 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:09.049 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 25769803801 2024-04-25T09:01:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803801 2024-04-25T09:01:09.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2024-04-25T09:01:09.050 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:01:09.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803799 -lt 25769803801 2024-04-25T09:01:09.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:01:10.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:01:10.446 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:01:10.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803802 -lt 25769803801 2024-04-25T09:01:10.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:10.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:10.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607570 2024-04-25T09:01:10.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:01:10.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:10.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607570 2024-04-25T09:01:10.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607570 2024-04-25T09:01:10.867 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607570 2024-04-25T09:01:10.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607570' 2024-04-25T09:01:10.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:01:11.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607570 -lt 51539607570 2024-04-25T09:01:11.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:11.339 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378634 2024-04-25T09:01:11.340 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:11.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:01:11.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378634 2024-04-25T09:01:11.342 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:11.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378634 2024-04-25T09:01:11.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378634' 2024-04-25T09:01:11.344 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 81604378634 2024-04-25T09:01:11.344 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:01:11.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378634 -lt 81604378634 2024-04-25T09:01:11.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:01:11.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:11.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:12.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:01:12.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:12.336 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:12.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:12.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:12.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:12.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:12.337 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:12.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:01:12.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:12.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:12.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:13.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:01:13.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:01:13.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:01:13.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:355: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:01:13.695 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.533600+0000 0'0 24:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:00:26.062882+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.533534+0000 0'0 24:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:06:07.090508+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.951264+0000 0'0 24:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:41:19.648658+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.533590+0000 0'0 24:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:52:23.432002+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:08.322263+0000 0'0 24:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:01:06.649651+0000 0'0 2024-04-25T09:01:06.649651+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:58:39.083163+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.457865+0000 0'0 24:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:26:30.405159+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:01.013324+0000 0'0 23:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:14:31.211976+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.883168+0000 0'0 24:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:44:52.643342+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.533924+0000 0'0 24:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:28:00.570186+0000 0 0 2024-04-25T09:01:13.696 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T09:01:13.697 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:01:13.697 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T09:01:13.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:357: TEST_rep_recovery_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-04-25T09:01:13.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:13.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:01:13.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:13.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: get_osds test-pool myobject 2024-04-25T09:01:13.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:01:13.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-25T09:01:13.875 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-04-25T09:01:13.875 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:01:14.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:01:14.293 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:01:14.293 INFO:tasks.workunit.client.0.smithi088.stderr:0' 2024-04-25T09:01:14.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-25T09:01:14.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: initial_osds=('2' '1' '0') 2024-04-25T09:01:14.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: local -a initial_osds 2024-04-25T09:01:14.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:360: TEST_rep_recovery_unfound: local last_osd=0 2024-04-25T09:01:14.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:361: TEST_rep_recovery_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-04-25T09:01:14.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:01:14.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:01:14.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:01:14.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:01:14.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:01:14.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:01:14.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:362: TEST_rep_recovery_unfound: ceph osd down 0 2024-04-25T09:01:15.672 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already down. 2024-04-25T09:01:15.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:363: TEST_rep_recovery_unfound: ceph osd out 0 2024-04-25T09:01:16.255 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. 2024-04-25T09:01:16.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:365: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:01:16.650 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:01:16.650 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.533600+0000 0'0 24:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:00:26.062882+0000 0 0 2024-04-25T09:01:16.650 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.533534+0000 0'0 24:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:06:07.090508+0000 0 0 2024-04-25T09:01:16.650 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:01:14.638274+0000 0'0 25:20 [2,1] 2 [2,1] 2 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:01:16.650 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.533590+0000 0'0 24:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:52:23.432002+0000 0 0 2024-04-25T09:01:16.650 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 peering 2024-04-25T09:01:14.639926+0000 24'1 25:16 [2,1] 2 [2,1] 2 0'0 2024-04-25T09:01:06.649651+0000 0'0 2024-04-25T09:01:06.649651+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:01:16.650 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.457865+0000 0'0 24:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:26:30.405159+0000 0 0 2024-04-25T09:01:16.651 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-04-25T09:01:01.013324+0000 0'0 24:46 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:14:31.211976+0000 0 0 2024-04-25T09:01:16.651 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:01:14.636822+0000 0'0 25:20 [2,1] 2 [2,1] 2 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:01:16.651 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:00.533924+0000 0'0 24:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:28:00.570186+0000 0 0 2024-04-25T09:01:16.651 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T09:01:16.651 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:01:16.651 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T09:01:16.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:367: TEST_rep_recovery_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-04-25T09:01:16.666 INFO:tasks.workunit.client.0.smithi088.stderr:4+0 records in 2024-04-25T09:01:16.666 INFO:tasks.workunit.client.0.smithi088.stderr:4+0 records out 2024-04-25T09:01:16.666 INFO:tasks.workunit.client.0.smithi088.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000172228 s, 23.8 MB/s 2024-04-25T09:01:16.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:01:16.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:16.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:16.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:16.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:17.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:17.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:17.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:17.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:17.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:17.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:17.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:17.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:17.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:17.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:17.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:17.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:17.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:17.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:18.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:18.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:18.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:18.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:18.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:18.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:18.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:18.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:18.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:18.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:18.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:18.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:19.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:19.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:19.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:19.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:19.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:19.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:19.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:19.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:19.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:19.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:19.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:19.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:20.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:20.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:20.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:20.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:20.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:20.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:20.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:20.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:20.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:20.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:20.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:20.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:21.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:21.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:21.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:21.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:21.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:21.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:21.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:21.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:21.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:21.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:21.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:21.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:21.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:21.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:22.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:22.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:22.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:22.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:22.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:22.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:22.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:22.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:22.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:22.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:23.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:23.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:23.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:23.165 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:23.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:23.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:23.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:23.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:23.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:23.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:23.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:23.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:23.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:24.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:24.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:24.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:24.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:24.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:24.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:24.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:24.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:24.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:24.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:24.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:24.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:24.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:24.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:25.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:25.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:25.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:25.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:25.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:25.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:25.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:25.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:25.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:25.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:25.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:25.977 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:26.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:26.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:26.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:26.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:26.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:26.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:26.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:26.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:26.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:26.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:26.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:26.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:27.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:27.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:27.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:27.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:27.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:27.359 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:27.509 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:27.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:27.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:27.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:27.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:27.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:27.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:28.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:28.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:28.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:28.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:28.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:28.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:28.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:28.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:28.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:28.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:28.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:28.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:29.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:29.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:29.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:29.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:29.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:29.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:29.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:29.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:29.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:29.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:29.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:30.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:30.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:30.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:30.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:30.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:30.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:30.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:30.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:30.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:30.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:30.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:30.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:30.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:31.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:31.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:31.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:31.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:31.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:31.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:31.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:31.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:31.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:31.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:31.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:31.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:32.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:01:32.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:01:32.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:373: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 0 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj75 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:01:32.175 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj75 2024-04-25T09:01:32.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:01:32.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-25T09:01:32.176 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-04-25T09:01:32.176 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:01:32.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:01:32.666 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T09:01:32.666 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:01:32.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:01:32.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:01:32.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=2 2024-04-25T09:01:32.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:01:32.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:01:32.668 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-04-25T09:01:32.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:01:32.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-04-25T09:01:32.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:01:32.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-04-25T09:01:32.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:01:32.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:01:32.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:01:32.669 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-04-25T09:01:32.669 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:01:32.669 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:01:32.670 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:01:32.670 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:32.670 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T09:01:32.670 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.2.asok 2024-04-25T09:01:32.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:01:32.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:01:32.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:01:32.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:01:32.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.2 2024-04-25T09:01:32.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:01:32.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:01:32.788 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:01:32.788 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:32.788 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T09:01:32.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.2.asok 2024-04-25T09:01:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:01:32.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.2.asok injectdataerr test-pool obj75 2024-04-25T09:01:32.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:374: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 1 2024-04-25T09:01:32.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:01:32.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:01:32.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj75 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj75 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:01:32.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-25T09:01:32.908 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-04-25T09:01:32.908 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:01:33.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:01:33.299 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T09:01:33.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:01:33.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:01:33.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:01:33.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:01:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:01:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:01:33.300 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:01:33.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:01:33.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:01:33.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:01:33.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:01:33.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:01:33.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:01:33.302 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:01:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:01:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:01:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:01:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:01:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T09:01:33.302 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T09:01:33.303 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.389694/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:01:33.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:01:33.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:01:33.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:01:33.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:01:33.412 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:01:33.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:01:33.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:01:33.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:33.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T09:01:33.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.389694/ceph-osd.1.asok 2024-04-25T09:01:33.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:01:33.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.389694/ceph-osd.1.asok injectdataerr test-pool obj75 2024-04-25T09:01:33.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:376: TEST_rep_recovery_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:01:33.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:01:33.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:01:33.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:01:33.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:01:33.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:33.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:01:33.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:01:33.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:01:33.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:01:33.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:01:33.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:01:33.541 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:01:33.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:01:33.542 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-04-25T09:01:33.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=216e48ad-ddcc-4ac6-8ab9-7bf658b33b37 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.389694/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:01:33.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:01:33.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:01:33.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:01:33.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:01:33.575 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:01:33.582+0000 7f3af2b437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:33.577 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:01:33.586+0000 7f3af2b437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:33.579 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:01:33.586+0000 7f3af2b437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:33.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:01:33.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:01:33.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:01:33.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:01:33.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:01:33.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:33.973 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:01:33.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:01:33.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:33.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:34.137 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:01:34.146+0000 7f3af2b437c0 -1 Falling back to public interface 2024-04-25T09:01:34.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:35.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:35.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:35.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:01:35.429 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:01:35.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:35.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:35.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:36.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:36.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:36.843 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:01:36.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:01:36.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:36.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:37.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:37.375 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:01:37.382+0000 7f3af2b437c0 -1 osd.0 24 log_to_monitors true 2024-04-25T09:01:38.277 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:38.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:38.278 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:01:38.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:01:38.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:38.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:39.018 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up out weight 0 up_from 30 up_thru 19 down_at 25 last_clean_interval [6,24) [v2:127.0.0.1:6802/3100747978,v1:127.0.0.1:6803/3100747978] [v2:127.0.0.1:6804/3100747978,v1:127.0.0.1:6805/3100747978] exists,up 44a04b79-a56e-43a6-bdf7-57e9eb6f7d03 2024-04-25T09:01:39.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:01:39.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:01:39.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:01:39.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:377: TEST_rep_recovery_unfound: ceph osd in 0 2024-04-25T09:01:40.097 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already in. 2024-04-25T09:01:40.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:379: TEST_rep_recovery_unfound: sleep 15 2024-04-25T09:01:55.116 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:01:55.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2024-04-25T09:01:55.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2024-04-25T09:01:55.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:01:55.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:01:55.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:01:55.118 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:01:55.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovery_unfound+degraded 2024-04-25T09:01:55.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovery_unfound+degraded 2024-04-25T09:01:55.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2024-04-25T09:01:55.540 INFO:tasks.workunit.client.0.smithi088.stdout:active+recovery_unfound+degraded 2024-04-25T09:01:55.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 0 = 0 ']' 2024-04-25T09:01:55.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:385: TEST_rep_recovery_unfound: break 2024-04-25T09:01:55.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:391: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:01:55.932 INFO:tasks.workunit.client.0.smithi088.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:01:55.932 INFO:tasks.workunit.client.0.smithi088.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:40.943182+0000 0'0 38:78 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:42:24.437495+0000 0 0 2024-04-25T09:01:55.932 INFO:tasks.workunit.client.0.smithi088.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:42.083352+0000 0'0 38:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:11:06.622940+0000 0 0 2024-04-25T09:01:55.932 INFO:tasks.workunit.client.0.smithi088.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:42.085800+0000 0'0 38:50 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:49:03.875428+0000 0 0 2024-04-25T09:01:55.932 INFO:tasks.workunit.client.0.smithi088.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:42.083158+0000 0'0 38:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:18:19.062898+0000 0 0 2024-04-25T09:01:55.932 INFO:tasks.workunit.client.0.smithi088.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-04-25T09:01:46.403464+0000 28'101 38:347 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:01:06.649651+0000 0'0 2024-04-25T09:01:06.649651+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:01:55.932 INFO:tasks.workunit.client.0.smithi088.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:41.034518+0000 0'0 38:78 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:55:00.546170+0000 0 0 2024-04-25T09:01:55.933 INFO:tasks.workunit.client.0.smithi088.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:42.084424+0000 0'0 38:63 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:45:32.151868+0000 0 0 2024-04-25T09:01:55.933 INFO:tasks.workunit.client.0.smithi088.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:42.085399+0000 0'0 38:57 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:24:54.389941+0000 0 0 2024-04-25T09:01:55.933 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:01:42.083497+0000 0'0 38:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T08:59:09.457510+0000 0'0 2024-04-25T08:59:09.457510+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:22:21.493818+0000 0 0 2024-04-25T09:01:55.933 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T09:01:55.933 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:01:55.933 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pgs 2024-04-25T09:01:55.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: ceph pg 2.0 list_unfound 2024-04-25T09:01:55.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: grep -q obj75 2024-04-25T09:01:56.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:395: TEST_rep_recovery_unfound: timeout 5 rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:01.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:396: TEST_rep_recovery_unfound: test 124 = 124 2024-04-25T09:02:01.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:398: TEST_rep_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-25T09:02:01.392 INFO:tasks.workunit.client.0.smithi088.stderr:pg has no unfound objects 2024-04-25T09:02:01.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:400: TEST_rep_recovery_unfound: wait_for_clean 2024-04-25T09:02:01.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:02:01.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:02:01.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:02:01.413 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:02:01.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:02:01.413 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:02:01.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:02:01.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:02:01.414 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:02:01.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:02:01.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:02:01.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:02:01.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:02:01.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:02:01.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:02:01.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:02:01.959 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:02:01.959 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:02:01.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:02:01.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:02:01.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:02:02.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018887 2024-04-25T09:02:02.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018887 2024-04-25T09:02:02.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887' 2024-04-25T09:02:02.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:02:02.117 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:02:02.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607583 2024-04-25T09:02:02.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607583 2024-04-25T09:02:02.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887 1-51539607583' 2024-04-25T09:02:02.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:02:02.273 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:02:02.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378647 2024-04-25T09:02:02.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378647 2024-04-25T09:02:02.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887 1-51539607583 2-81604378647' 2024-04-25T09:02:02.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:02:02.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018887 2024-04-25T09:02:02.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:02:02.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:02:02.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018887 2024-04-25T09:02:02.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:02:02.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018887 2024-04-25T09:02:02.435 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 128849018887 2024-04-25T09:02:02.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018887' 2024-04-25T09:02:02.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:02:02.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018885 -lt 128849018887 2024-04-25T09:02:02.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:02:03.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:02:03.852 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:02:04.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018888 -lt 128849018887 2024-04-25T09:02:04.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:02:04.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607583 2024-04-25T09:02:04.279 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:02:04.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:02:04.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607583 2024-04-25T09:02:04.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:02:04.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607583 2024-04-25T09:02:04.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607583' 2024-04-25T09:02:04.283 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 51539607583 2024-04-25T09:02:04.283 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:02:04.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607583 -lt 51539607583 2024-04-25T09:02:04.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:02:04.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378647 2024-04-25T09:02:04.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:02:04.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:02:04.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378647 2024-04-25T09:02:04.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:02:04.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378647 2024-04-25T09:02:04.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378647' 2024-04-25T09:02:04.717 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 81604378647 2024-04-25T09:02:04.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:02:05.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378647 -lt 81604378647 2024-04-25T09:02:05.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:02:05.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:02:05.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:02:05.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:02:05.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:02:05.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:02:05.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:02:05.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:02:05.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:02:05.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:02:05.791 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:02:06.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:02:06.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:02:06.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:02:06.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:02:06.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:02:06.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:02:06.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:02:06.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:02:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj1 = obj75 ']' 2024-04-25T09:02:06.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:06.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj2 = obj75 ']' 2024-04-25T09:02:06.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:06.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj3 = obj75 ']' 2024-04-25T09:02:06.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:06.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj4 = obj75 ']' 2024-04-25T09:02:06.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:06.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj5 = obj75 ']' 2024-04-25T09:02:06.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:06.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj6 = obj75 ']' 2024-04-25T09:02:06.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:06.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:06.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj7 = obj75 ']' 2024-04-25T09:02:06.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj8 = obj75 ']' 2024-04-25T09:02:07.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj9 = obj75 ']' 2024-04-25T09:02:07.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj10 = obj75 ']' 2024-04-25T09:02:07.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj11 = obj75 ']' 2024-04-25T09:02:07.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj12 = obj75 ']' 2024-04-25T09:02:07.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj13 = obj75 ']' 2024-04-25T09:02:07.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj14 = obj75 ']' 2024-04-25T09:02:07.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj15 = obj75 ']' 2024-04-25T09:02:07.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj16 = obj75 ']' 2024-04-25T09:02:07.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj17 = obj75 ']' 2024-04-25T09:02:07.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj18 = obj75 ']' 2024-04-25T09:02:07.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj19 = obj75 ']' 2024-04-25T09:02:07.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj20 = obj75 ']' 2024-04-25T09:02:07.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj21 = obj75 ']' 2024-04-25T09:02:07.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj22 = obj75 ']' 2024-04-25T09:02:07.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj23 = obj75 ']' 2024-04-25T09:02:07.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj24 = obj75 ']' 2024-04-25T09:02:07.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj25 = obj75 ']' 2024-04-25T09:02:07.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj26 = obj75 ']' 2024-04-25T09:02:07.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj27 = obj75 ']' 2024-04-25T09:02:07.595 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj28 = obj75 ']' 2024-04-25T09:02:07.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj29 = obj75 ']' 2024-04-25T09:02:07.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj30 = obj75 ']' 2024-04-25T09:02:07.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj31 = obj75 ']' 2024-04-25T09:02:07.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj32 = obj75 ']' 2024-04-25T09:02:07.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj33 = obj75 ']' 2024-04-25T09:02:07.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj34 = obj75 ']' 2024-04-25T09:02:07.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj35 = obj75 ']' 2024-04-25T09:02:07.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj36 = obj75 ']' 2024-04-25T09:02:07.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj37 = obj75 ']' 2024-04-25T09:02:07.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj38 = obj75 ']' 2024-04-25T09:02:07.953 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:07.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:07.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj39 = obj75 ']' 2024-04-25T09:02:07.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj40 = obj75 ']' 2024-04-25T09:02:08.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj41 = obj75 ']' 2024-04-25T09:02:08.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj42 = obj75 ']' 2024-04-25T09:02:08.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj43 = obj75 ']' 2024-04-25T09:02:08.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj44 = obj75 ']' 2024-04-25T09:02:08.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj45 = obj75 ']' 2024-04-25T09:02:08.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj46 = obj75 ']' 2024-04-25T09:02:08.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj47 = obj75 ']' 2024-04-25T09:02:08.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj48 = obj75 ']' 2024-04-25T09:02:08.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj49 = obj75 ']' 2024-04-25T09:02:08.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj50 = obj75 ']' 2024-04-25T09:02:08.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj51 = obj75 ']' 2024-04-25T09:02:08.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj52 = obj75 ']' 2024-04-25T09:02:08.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj53 = obj75 ']' 2024-04-25T09:02:08.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj54 = obj75 ']' 2024-04-25T09:02:08.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj55 = obj75 ']' 2024-04-25T09:02:08.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj56 = obj75 ']' 2024-04-25T09:02:08.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj57 = obj75 ']' 2024-04-25T09:02:08.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj58 = obj75 ']' 2024-04-25T09:02:08.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj59 = obj75 ']' 2024-04-25T09:02:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj60 = obj75 ']' 2024-04-25T09:02:08.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj61 = obj75 ']' 2024-04-25T09:02:08.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj62 = obj75 ']' 2024-04-25T09:02:08.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.769 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj63 = obj75 ']' 2024-04-25T09:02:08.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj64 = obj75 ']' 2024-04-25T09:02:08.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj65 = obj75 ']' 2024-04-25T09:02:08.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj66 = obj75 ']' 2024-04-25T09:02:08.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj67 = obj75 ']' 2024-04-25T09:02:08.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj68 = obj75 ']' 2024-04-25T09:02:08.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj69 = obj75 ']' 2024-04-25T09:02:08.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:08.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:08.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj70 = obj75 ']' 2024-04-25T09:02:08.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj71 = obj75 ']' 2024-04-25T09:02:09.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj72 = obj75 ']' 2024-04-25T09:02:09.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.067 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj73 = obj75 ']' 2024-04-25T09:02:09.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj74 = obj75 ']' 2024-04-25T09:02:09.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj75 = obj75 ']' 2024-04-25T09:02:09.128 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:406: TEST_rep_recovery_unfound: rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.154 INFO:tasks.workunit.client.0.smithi088.stderr:error getting test-pool/obj75: (2) No such file or directory 2024-04-25T09:02:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj76 = obj75 ']' 2024-04-25T09:02:09.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj77 = obj75 ']' 2024-04-25T09:02:09.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj78 = obj75 ']' 2024-04-25T09:02:09.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj79 = obj75 ']' 2024-04-25T09:02:09.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj80 = obj75 ']' 2024-04-25T09:02:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.303 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj81 = obj75 ']' 2024-04-25T09:02:09.304 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj82 = obj75 ']' 2024-04-25T09:02:09.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj83 = obj75 ']' 2024-04-25T09:02:09.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj84 = obj75 ']' 2024-04-25T09:02:09.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj85 = obj75 ']' 2024-04-25T09:02:09.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj86 = obj75 ']' 2024-04-25T09:02:09.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj87 = obj75 ']' 2024-04-25T09:02:09.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj88 = obj75 ']' 2024-04-25T09:02:09.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj89 = obj75 ']' 2024-04-25T09:02:09.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj90 = obj75 ']' 2024-04-25T09:02:09.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj91 = obj75 ']' 2024-04-25T09:02:09.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj92 = obj75 ']' 2024-04-25T09:02:09.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj93 = obj75 ']' 2024-04-25T09:02:09.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj94 = obj75 ']' 2024-04-25T09:02:09.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj95 = obj75 ']' 2024-04-25T09:02:09.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.747 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj96 = obj75 ']' 2024-04-25T09:02:09.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj97 = obj75 ']' 2024-04-25T09:02:09.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj98 = obj75 ']' 2024-04-25T09:02:09.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj99 = obj75 ']' 2024-04-25T09:02:09.837 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:02:09.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj100 = obj75 ']' 2024-04-25T09:02:09.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:413: TEST_rep_recovery_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:02:09.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:415: TEST_rep_recovery_unfound: delete_pool test-pool 2024-04-25T09:02:09.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-04-25T09:02:09.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-04-25T09:02:10.380 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test-pool' does not exist 2024-04-25T09:02:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:02:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:02:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:02:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:02:10.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:02:10.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:02:10.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:02:10.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:02:10.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:02:10.706 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:02:10.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:02:10.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:02:10.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:02:10.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:02:10.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:02:10.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:02:10.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:02:10.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:02:10.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:02:10.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:02:10.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:02:10.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:02:10.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:02:10.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:02:10.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:10.744 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T09:02:10.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T09:02:10.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:02:10.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:02:10.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:02:10.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-rep-recov-eio.sh 0 2024-04-25T09:02:10.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:02:10.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:02:10.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:02:10.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:02:10.746 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:02:10.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:02:10.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:02:10.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:02:10.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:02:10.749 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:02:10.750 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:02:10.750 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:02:10.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:02:10.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:02:10.751 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:02:10.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:02:10.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:02:10.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:02:10.753 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:02:10.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:02:10.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:02:10.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:02:10.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:02:10.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:10.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.389694 2024-04-25T09:02:10.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.389694 2024-04-25T09:02:10.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:02:10.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:02:10.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:02:10.758 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:02:10.758 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:02:10.813 INFO:tasks.workunit:Running workunit osd/osd-reuse-id.sh... 2024-04-25T09:02:10.814 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/osd-reuse-id.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2024-04-25T09:02:10.867 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:02:10.872 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:02:10.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:02:10.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:02:10.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:02:10.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:02:10.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-reuse-id 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:20: run: local dir=td/osd-reuse-id 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:21: run: shift 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: export CEPH_MON=127.0.0.1:7123 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: CEPH_MON=127.0.0.1:7123 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:24: run: export CEPH_ARGS 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: uuidgen 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: CEPH_ARGS+='--fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none ' 2024-04-25T09:02:10.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7123 ' 2024-04-25T09:02:10.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: set 2024-04-25T09:02:10.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:02:10.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: local funcs=TEST_reuse_id 2024-04-25T09:02:10.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:29: run: for func in $funcs 2024-04-25T09:02:10.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:30: run: setup td/osd-reuse-id 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reuse-id 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reuse-id 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:02:10.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:02:10.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:02:10.877 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:02:10.878 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:02:10.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:02:10.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:02:10.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:02:10.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:02:10.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:02:10.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:02:10.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:02:10.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:02:10.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:02:10.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:02:10.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:02:10.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:02:10.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:10.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:02:10.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.445638 2024-04-25T09:02:10.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:02:10.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:02:10.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reuse-id 2024-04-25T09:02:10.886 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:02:10.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:10.887 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:02:10.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.445638 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reuse-id 1' TERM HUP INT 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:31: run: TEST_reuse_id td/osd-reuse-id 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:37: TEST_reuse_id: local dir=td/osd-reuse-id 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:39: TEST_reuse_id: run_mon td/osd-reuse-id a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:02:10.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reuse-id 2024-04-25T09:02:10.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:02:10.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:02:10.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:02:10.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reuse-id/a 2024-04-25T09:02:10.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reuse-id/a --run-dir=td/osd-reuse-id --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:02:11.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:02:11.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:02:11.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:02:11.362 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:02:11.362 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:11.362 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:02:11.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.445638/$cluster-$name.asok' 2024-04-25T09:02:11.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reuse-id/a '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reuse-id/log --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:02:11.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:02:11.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:02:11.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:02:11.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:02:11.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:02:11.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:02:11.559 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:02:11.559 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:02:11.559 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:02:11.560 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:02:11.560 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:11.560 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:02:11.560 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.445638/ceph-mon.a.asok 2024-04-25T09:02:11.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:02:11.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.445638/ceph-mon.a.asok config get fsid 2024-04-25T09:02:11.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:02:11.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:02:11.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:02:11.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:02:11.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:02:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:02:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:02:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:02:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:02:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:02:11.673 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.445638/ceph-mon.a.asok 2024-04-25T09:02:11.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:02:11.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.445638/ceph-mon.a.asok config get mon_host 2024-04-25T09:02:11.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:40: TEST_reuse_id: run_mgr td/osd-reuse-id x 2024-04-25T09:02:11.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reuse-id 2024-04-25T09:02:11.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:02:11.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:02:11.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:02:11.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reuse-id/x 2024-04-25T09:02:11.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:02:12.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:02:12.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:02:12.687 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:02:12.687 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:02:12.688 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:12.688 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:02:12.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.445638/$cluster-$name.asok' 2024-04-25T09:02:12.688 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:02:12.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reuse-id/x '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:41: TEST_reuse_id: run_osd td/osd-reuse-id 0 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/0 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:02:12.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/0' 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/0/journal' 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.445638/$cluster-$name.asok' 2024-04-25T09:02:12.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:02:12.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/0 2024-04-25T09:02:12.728 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:02:12.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=095b82d9-6485-4bfb-99b5-df149e868f17 2024-04-25T09:02:12.728 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 095b82d9-6485-4bfb-99b5-df149e868f17 2024-04-25T09:02:12.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 095b82d9-6485-4bfb-99b5-df149e868f17' 2024-04-25T09:02:12.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:02:12.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUHCpmF8ITLRAA/IGlmzRkF3BGb6yyYFyp/g== 2024-04-25T09:02:12.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUHCpmF8ITLRAA/IGlmzRkF3BGb6yyYFyp/g=="}' 2024-04-25T09:02:12.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 095b82d9-6485-4bfb-99b5-df149e868f17 -i td/osd-reuse-id/0/new.json 2024-04-25T09:02:13.068 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:02:13.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/0/new.json 2024-04-25T09:02:13.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUHCpmF8ITLRAA/IGlmzRkF3BGb6yyYFyp/g== --osd-uuid 095b82d9-6485-4bfb-99b5-df149e868f17 2024-04-25T09:02:13.111 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:13.122+0000 7f681d6b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:13.145 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:13.158+0000 7f681d6b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:13.146 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:13.158+0000 7f681d6b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:13.147 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:13.158+0000 7f681d6b47c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:02:13.147 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:13.158+0000 7f681d6b47c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:02:13.147 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:13.158+0000 7f681d6b47c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:02:13.185 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:13.198+0000 7f681d6b47c0 -1 bluestore(td/osd-reuse-id/0) _read_fsid unparsable uuid 2024-04-25T09:02:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/0/keyring 2024-04-25T09:02:19.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:02:19.455 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:02:19.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:02:19.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:02:19.988 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:02:19.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:02:19.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:02:19.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:02:19.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:02:19.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:02:20.011 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:20.022+0000 7f6a5e58e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:20.013 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:20.026+0000 7f6a5e58e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:20.014 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:20.026+0000 7f6a5e58e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:20.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:02:20.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:02:20.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:02:20.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:02:20.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:02:20.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:20.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:02:20.432 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:02:20.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:20.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:20.585 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:20.598+0000 7f6a5e58e7c0 -1 Falling back to public interface 2024-04-25T09:02:20.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:21.876 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:02:21.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:21.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:21.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:02:21.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:21.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:22.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:23.266 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:23.277+0000 7f6a5e58e7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:02:23.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:23.280 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:02:23.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:23.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:02:23.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:23.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:23.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:24.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:24.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:24.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:02:24.910 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:02:24.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:24.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:25.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:26.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:26.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:26.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:02:26.365 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:02:26.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:26.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:26.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:27.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:27.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:27.803 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:02:27.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:02:27.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:27.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:28.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:29.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:29.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:29.325 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:02:29.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:02:29.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:29.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:29.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:30.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:30.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:30.748 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:02:30.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:02:30.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:30.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:31.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:32.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:32.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:32.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:02:32.212 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:02:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:32.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:32.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:33.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:33.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:33.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:02:33.660 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:02:33.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:33.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:34.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:35.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:35.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:35.124 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:02:35.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:02:35.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:35.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:35.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:36.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:36.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:36.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:02:36.529 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:02:36.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:36.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:36.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:37.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:37.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:37.981 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:02:37.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:02:37.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:37.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:38.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:39.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:39.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:39.814 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:02:39.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:02:39.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:39.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:40.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:41.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:41.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:41.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:02:41.753 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:02:41.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:41.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:43.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:44.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:44.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:44.032 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:02:44.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:02:44.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:44.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:44.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:45.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:45.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:45.600 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:02:45.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:02:45.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:45.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:46.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:47.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:47.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:47.454 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:02:47.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:02:47.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:47.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:47.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:48.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:48.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:48.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:02:48.915 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:02:48.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:48.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:49.326 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2703690242,v1:127.0.0.1:6803/2703690242] [v2:127.0.0.1:6804/2703690242,v1:127.0.0.1:6805/2703690242] exists,up 095b82d9-6485-4bfb-99b5-df149e868f17 2024-04-25T09:02:49.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:02:49.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:42: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-04-25T09:02:49.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.445638/$cluster-$name.asok' 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:02:49.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:02:49.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-04-25T09:02:49.334 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:02:49.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1fa6d8e-2f0c-4f4f-b6c5-61e042f522df 2024-04-25T09:02:49.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a1fa6d8e-2f0c-4f4f-b6c5-61e042f522df' 2024-04-25T09:02:49.335 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 a1fa6d8e-2f0c-4f4f-b6c5-61e042f522df 2024-04-25T09:02:49.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:02:49.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5HCpmmB7MFRAAz1pOHHvzvDlmaQKu5A+GOw== 2024-04-25T09:02:49.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5HCpmmB7MFRAAz1pOHHvzvDlmaQKu5A+GOw=="}' 2024-04-25T09:02:49.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1fa6d8e-2f0c-4f4f-b6c5-61e042f522df -i td/osd-reuse-id/1/new.json 2024-04-25T09:02:49.849 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:02:49.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-04-25T09:02:49.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5HCpmmB7MFRAAz1pOHHvzvDlmaQKu5A+GOw== --osd-uuid a1fa6d8e-2f0c-4f4f-b6c5-61e042f522df 2024-04-25T09:02:49.898 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:49.909+0000 7f5c982817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:49.933 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:49.945+0000 7f5c982817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:49.934 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:49.945+0000 7f5c982817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:49.935 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:49.945+0000 7f5c982817c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:02:49.935 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:49.945+0000 7f5c982817c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:02:49.935 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:49.945+0000 7f5c982817c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:02:49.972 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:49.985+0000 7f5c982817c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-04-25T09:02:56.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-04-25T09:02:56.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:02:56.818 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T09:02:56.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:02:56.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:02:57.414 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:02:57.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:02:57.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:02:57.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:02:57.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:02:57.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:02:57.443 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:57.453+0000 7f6f350d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:57.445 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:57.457+0000 7f6f350d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:57.446 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:57.457+0000 7f6f350d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:57.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:02:57.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:02:57.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:02:57.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:02:57.821 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:02:57.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:02:57.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:57.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:02:57.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:57.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:57.997 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:02:58.009+0000 7f6f350d17c0 -1 Falling back to public interface 2024-04-25T09:02:58.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:59.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:59.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:59.212 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:02:59.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:02:59.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:59.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:59.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:00.602 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:00.613+0000 7f6f350d17c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:03:00.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:00.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:00.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:03:00.617 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:03:00.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:00.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:01.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:02.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:02.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:02.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:03:02.413 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:03:02.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:02.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:02.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:03.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:03.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:03.869 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:03:03.870 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:03:03.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:03.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:04.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:05.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:05.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:05.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:03:05.317 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:03:05.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:05.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:06.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:07.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:07.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:07.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:03:07.054 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:03:07.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:07.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:07.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:08.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:08.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:08.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:03:08.713 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:03:08.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:08.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:09.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:10.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:10.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:10.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:03:10.163 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:03:10.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:10.163 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:10.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:11.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:11.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:11.634 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:03:11.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:03:11.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:11.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:12.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:13.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:13.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:13.191 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:03:13.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:03:13.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:13.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:13.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:14.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:14.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:14.774 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:03:14.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:03:14.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:14.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:15.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:16.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:16.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:16.200 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:03:16.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:03:16.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:16.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:17.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:18.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:18.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:18.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:03:18.118 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:03:18.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:18.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:18.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:19.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:19.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:19.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:03:19.944 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:03:19.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:19.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:21.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:22.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:03:22.074 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:03:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:22.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:23.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:24.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:24.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:24.236 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:03:24.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:03:24.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:24.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:24.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:25.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:25.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:25.933 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:03:25.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:03:25.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:25.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:26.401 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1528557316,v1:127.0.0.1:6811/1528557316] [v2:127.0.0.1:6812/1528557316,v1:127.0.0.1:6813/1528557316] exists,up a1fa6d8e-2f0c-4f4f-b6c5-61e042f522df 2024-04-25T09:03:26.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:03:26.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:03:26.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:03:26.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:43: TEST_reuse_id: create_rbd_pool 2024-04-25T09:03:26.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T09:03:26.873 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' does not exist 2024-04-25T09:03:26.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T09:03:26.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T09:03:27.534 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'rbd' already exists 2024-04-25T09:03:27.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:03:28.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T09:03:30.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:44: TEST_reuse_id: wait_for_clean 2024-04-25T09:03:30.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:03:30.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:03:30.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:03:30.427 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:03:30.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:03:30.428 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:03:30.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:03:30.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:03:30.428 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:03:30.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:03:30.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:03:30.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:03:30.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:03:30.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:03:30.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:03:30.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:03:30.961 INFO:tasks.workunit.client.0.smithi088.stderr:1' 2024-04-25T09:03:30.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:03:30.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:30.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:03:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836494 2024-04-25T09:03:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836494 2024-04-25T09:03:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494' 2024-04-25T09:03:31.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:03:31.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:03:31.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672966 2024-04-25T09:03:31.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672966 2024-04-25T09:03:31.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494 1-42949672966' 2024-04-25T09:03:31.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:31.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836494 2024-04-25T09:03:31.295 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:31.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:03:31.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:31.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836494 2024-04-25T09:03:31.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836494 2024-04-25T09:03:31.298 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836494 2024-04-25T09:03:31.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-04-25T09:03:31.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:03:31.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836494 2024-04-25T09:03:31.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:03:31.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672966 2024-04-25T09:03:31.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:03:31.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:03:31.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672966 2024-04-25T09:03:31.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:03:31.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672966 2024-04-25T09:03:31.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-04-25T09:03:31.735 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672966 2024-04-25T09:03:31.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:03:32.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672964 -lt 42949672966 2024-04-25T09:03:32.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:03:33.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:03:33.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:03:33.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672967 -lt 42949672966 2024-04-25T09:03:33.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:03:33.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:03:33.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:03:34.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-25T09:03:34.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:03:34.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:03:34.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:03:34.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:03:34.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:03:34.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:03:34.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:03:34.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-25T09:03:34.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:03:34.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:03:34.520 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:03:35.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-25T09:03:35.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:03:35.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:03:35.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:45: TEST_reuse_id: destroy_osd td/osd-reuse-id 1 2024-04-25T09:03:35.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/osd-reuse-id 2024-04-25T09:03:35.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=1 2024-04-25T09:03:35.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.1 2024-04-25T09:03:35.738 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already out. 2024-04-25T09:03:35.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/osd-reuse-id TERM osd.1 2024-04-25T09:03:35.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:03:35.756 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:03:35.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:03:35.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:03:35.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:03:36.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:03:36.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.1 2024-04-25T09:03:36.484 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 is already down. 2024-04-25T09:03:36.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.1 --yes-i-really-mean-it 2024-04-25T09:03:36.985 INFO:tasks.workunit.client.0.smithi088.stderr:osd.1 does not exist 2024-04-25T09:03:37.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/osd-reuse-id/1 2024-04-25T09:03:37.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id/1 2024-04-25T09:03:37.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:03:37.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id/1 KILL 2024-04-25T09:03:37.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:03:37.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:03:37.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:03:37.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:03:37.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:03:37.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:03:37.004 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:03:37.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:03:37.005 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:03:37.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:03:37.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:03:37.007 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:03:37.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:03:37.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:03:37.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:03:37.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:03:37.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:03:37.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:03:37.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id/1 2024-04-25T09:03:37.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:03:37.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:37.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:03:37.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.445638 2024-04-25T09:03:37.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:03:37.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:03:37.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/osd-reuse-id/1 2024-04-25T09:03:37.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:46: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-04-25T09:03:37.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:03:37.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:03:37.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:03:37.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:03:37.020 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.445638/$cluster-$name.asok' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:03:37.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:03:37.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:03:37.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:03:37.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:03:37.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-04-25T09:03:37.022 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:03:37.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=77efab48-aeee-4b55-87a3-15215cf13500 2024-04-25T09:03:37.023 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 77efab48-aeee-4b55-87a3-15215cf13500 2024-04-25T09:03:37.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 77efab48-aeee-4b55-87a3-15215cf13500' 2024-04-25T09:03:37.023 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:03:37.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpHCpmmJ3qAhAAxqynOT0Z0VLNtil+I03UzA== 2024-04-25T09:03:37.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpHCpmmJ3qAhAAxqynOT0Z0VLNtil+I03UzA=="}' 2024-04-25T09:03:37.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 77efab48-aeee-4b55-87a3-15215cf13500 -i td/osd-reuse-id/1/new.json 2024-04-25T09:03:37.518 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:03:37.534 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-04-25T09:03:37.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpHCpmmJ3qAhAAxqynOT0Z0VLNtil+I03UzA== --osd-uuid 77efab48-aeee-4b55-87a3-15215cf13500 2024-04-25T09:03:37.566 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:37.576+0000 7fed260e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:37.646 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:37.656+0000 7fed260e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:37.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:37.660+0000 7fed260e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:37.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:37.660+0000 7fed260e07c0 -1 asok(0x556e9583e000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.445638/ceph-osd.1.asok': (2) No such file or directory 2024-04-25T09:03:37.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:37.660+0000 7fed260e07c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:03:37.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:37.660+0000 7fed260e07c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:03:37.648 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:37.660+0000 7fed260e07c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:03:37.767 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:37.776+0000 7fed260e07c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-04-25T09:03:44.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-04-25T09:03:44.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:03:44.618 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:03:44.619 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T09:03:44.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:03:45.153 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:03:45.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:03:45.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=719191b9-6957-4274-8018-305514ce0dfc --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.445638/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:03:45.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:03:45.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:03:45.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:03:45.184 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:45.196+0000 7ff0599007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:45.218 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:45.228+0000 7ff0599007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:45.232+0000 7ff0599007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:45.220 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:45.232+0000 7ff0599007c0 -1 asok(0x55e698cfa000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.445638/ceph-osd.1.asok': (2) No such file or directory 2024-04-25T09:03:45.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:03:45.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:03:45.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:03:45.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:03:45.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:03:45.611 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:03:45.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:45.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:03:45.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:45.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:45.785 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:45.796+0000 7ff0599007c0 -1 Falling back to public interface 2024-04-25T09:03:46.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:47.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:47.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:47.070 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:03:47.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:03:47.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:47.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:47.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:48.460 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:48.472+0000 7ff0599007c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:03:48.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:48.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:48.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:03:48.486 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:03:48.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:48.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:49.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:50.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:50.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:50.112 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:03:50.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:03:50.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:50.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:03:50.609 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3671752679,v1:127.0.0.1:6811/3671752679] [v2:127.0.0.1:6812/3671752679,v1:127.0.0.1:6813/3671752679] exists,up 77efab48-aeee-4b55-87a3-15215cf13500 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:32: run: teardown td/osd-reuse-id 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:03:50.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:03:50.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:03:50.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:03:50.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:03:50.718 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:03:50.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:03:50.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:03:50.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:03:50.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:03:50.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:03:50.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:03:50.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:03:50.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:03:50.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:03:50.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:03:50.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:03:50.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:50.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:03:50.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.445638 2024-04-25T09:03:50.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:03:50.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-reuse-id 0 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:03:50.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:03:50.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:03:50.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:03:50.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:03:50.736 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:03:50.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:03:50.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:03:50.737 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:03:50.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:03:50.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:03:50.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:03:50.739 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:03:50.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:03:50.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:03:50.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:03:50.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:03:50.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:50.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.445638 2024-04-25T09:03:50.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.445638 2024-04-25T09:03:50.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:03:50.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:03:50.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:03:50.744 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:03:50.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:03:50.796 INFO:tasks.workunit:Running workunit osd/pg-split-merge.sh... 2024-04-25T09:03:50.797 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/pg-split-merge.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2024-04-25T09:03:50.850 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:03:50.855 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:03:50.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:03:50.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:03:50.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:03:50.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:03:50.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:03:50.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/pg-split-merge 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:5: run: local dir=td/pg-split-merge 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:6: run: shift 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: export CEPH_MON=127.0.0.1:7147 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: CEPH_MON=127.0.0.1:7147 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:9: run: export CEPH_ARGS 2024-04-25T09:03:50.856 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: uuidgen 2024-04-25T09:03:50.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: CEPH_ARGS+='--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none ' 2024-04-25T09:03:50.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:03:50.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: set 2024-04-25T09:03:50.857 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: local 'funcs=TEST_a_merge_empty 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_import_after_merge_and_gap 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_import_after_split' 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:03:50.859 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:03:50.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:03:50.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:03:50.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:03:50.860 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:03:50.860 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:03:50.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:03:50.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:03:50.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:03:50.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:03:50.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:03:50.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:03:50.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:03:50.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:03:50.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:03:50.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:03:50.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:03:50.865 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:03:50.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:03:50.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:50.866 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:03:50.866 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451260 2024-04-25T09:03:50.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:03:50.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:03:50.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:03:50.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:03:50.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:50.868 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:03:50.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.451260 2024-04-25T09:03:50.870 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:03:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:03:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:03:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:03:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:03:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_a_merge_empty td/pg-split-merge 2024-04-25T09:03:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:22: TEST_a_merge_empty: local dir=td/pg-split-merge 2024-04-25T09:03:50.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:24: TEST_a_merge_empty: run_mon td/pg-split-merge a --osd_pool_default_size=3 2024-04-25T09:03:50.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:03:50.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:03:50.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:03:50.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:03:50.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:03:50.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=3 2024-04-25T09:03:51.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:03:51.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:03:51.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:03:51.384 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:03:51.384 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:51.384 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:03:51.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:03:51.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T09:03:51.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:03:51.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:03:51.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:03:51.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:03:51.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:03:51.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:03:51.606 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:03:51.606 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:03:51.606 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:03:51.606 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:51.606 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:51.606 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:03:51.607 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451260/ceph-mon.a.asok 2024-04-25T09:03:51.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:03:51.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451260/ceph-mon.a.asok config get fsid 2024-04-25T09:03:51.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:03:51.729 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:51.730 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:03:51.731 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451260/ceph-mon.a.asok 2024-04-25T09:03:51.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:03:51.731 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451260/ceph-mon.a.asok config get mon_host 2024-04-25T09:03:51.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:25: TEST_a_merge_empty: run_mgr td/pg-split-merge x 2024-04-25T09:03:51.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:03:51.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:03:51.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:03:51.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:03:51.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:03:51.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:03:52.758 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:03:52.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:03:52.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:03:52.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:03:52.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:52.759 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:03:52.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:03:52.760 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:03:52.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:03:52.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:26: TEST_a_merge_empty: run_osd td/pg-split-merge 0 2024-04-25T09:03:52.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:03:52.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:03:52.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:03:52.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:03:52.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:03:52.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:03:52.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:03:52.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:03:52.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:03:52.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:03:52.801 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:03:52.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:03:52.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6fae7448-ad9c-4076-bf12-2dd5212861ca 2024-04-25T09:03:52.804 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 6fae7448-ad9c-4076-bf12-2dd5212861ca 2024-04-25T09:03:52.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6fae7448-ad9c-4076-bf12-2dd5212861ca' 2024-04-25T09:03:52.804 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:03:52.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4HCpmnJ6XMRAAbYp+zNNCpPf4YTN8RWSluQ== 2024-04-25T09:03:52.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4HCpmnJ6XMRAAbYp+zNNCpPf4YTN8RWSluQ=="}' 2024-04-25T09:03:52.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6fae7448-ad9c-4076-bf12-2dd5212861ca -i td/pg-split-merge/0/new.json 2024-04-25T09:03:53.132 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:03:53.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:03:53.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4HCpmnJ6XMRAAbYp+zNNCpPf4YTN8RWSluQ== --osd-uuid 6fae7448-ad9c-4076-bf12-2dd5212861ca 2024-04-25T09:03:53.184 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:53.196+0000 7f42407b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:53.200 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:53.212+0000 7f42407b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:53.202 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:53.212+0000 7f42407b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:53.202 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:53.212+0000 7f42407b97c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:03:53.202 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:53.212+0000 7f42407b97c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:03:53.202 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:53.212+0000 7f42407b97c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:03:53.240 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:03:53.252+0000 7f42407b97c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:03:59.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:03:59.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:03:59.584 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:03:59.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:03:59.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:04:00.158 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:04:00.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:04:00.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:04:00.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:04:00.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:04:00.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:04:00.192 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:00.204+0000 7f07a293f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:00.232 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:00.244+0000 7f07a293f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:00.234 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:00.244+0000 7f07a293f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:00.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:04:00.609 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:04:00.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:04:00.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:04:00.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:04:00.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:00.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:04:00.610 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:04:00.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:00.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:00.797 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:00.808+0000 7f07a293f7c0 -1 Falling back to public interface 2024-04-25T09:04:01.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:02.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:02.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:02.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:04:02.051 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:04:02.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:02.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:02.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:03.374 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:03.383+0000 7f07a293f7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:04:03.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:03.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:03.466 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:04:03.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:04:03.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:03.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:04.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:05.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:05.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:05.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:04:05.102 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:04:05.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:05.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:05.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:06.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:06.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:06.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:04:06.549 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:04:06.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:06.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:07.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:08.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:08.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:08.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:04:08.021 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:04:08.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:08.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:08.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:09.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:09.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:09.586 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:04:09.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:04:09.586 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:09.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:10.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:11.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:11.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:11.021 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:04:11.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:04:11.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:11.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:11.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:12.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:12.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:12.563 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:04:12.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:04:12.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:12.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:12.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:14.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:14.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:14.001 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:04:14.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:04:14.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:14.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:14.526 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:15.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:15.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:15.528 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:04:15.529 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:04:15.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:15.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:15.955 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:16.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:16.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:16.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:04:16.958 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:04:16.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:16.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:17.472 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:18.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:18.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:18.474 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:04:18.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:04:18.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:18.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:19.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:20.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:20.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:20.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:04:20.191 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:04:20.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:20.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:20.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:21.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:21.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:21.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:04:21.853 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:04:21.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:21.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:22.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:23.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:23.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:23.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:04:23.947 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:04:23.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:23.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:24.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:25.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:25.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:25.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:04:25.889 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:04:25.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:25.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:26.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:27.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:27.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:27.422 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:04:27.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:04:27.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:27.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:04:27.925 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3298189445,v1:127.0.0.1:6803/3298189445] [v2:127.0.0.1:6804/3298189445,v1:127.0.0.1:6805/3298189445] exists,up 6fae7448-ad9c-4076-bf12-2dd5212861ca 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:27: TEST_a_merge_empty: run_osd td/pg-split-merge 1 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/1 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:04:27.930 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/1' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/1/journal' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:04:27.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/1 2024-04-25T09:04:27.932 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:04:27.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=009a2c57-76f8-4e70-9504-17bdf92177b0 2024-04-25T09:04:27.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 009a2c57-76f8-4e70-9504-17bdf92177b0' 2024-04-25T09:04:27.933 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 009a2c57-76f8-4e70-9504-17bdf92177b0 2024-04-25T09:04:27.933 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:04:27.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCbHCpm/3BOORAAwCoXa+BIOnWT+Cr/NNLzUw== 2024-04-25T09:04:27.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCbHCpm/3BOORAAwCoXa+BIOnWT+Cr/NNLzUw=="}' 2024-04-25T09:04:27.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 009a2c57-76f8-4e70-9504-17bdf92177b0 -i td/pg-split-merge/1/new.json 2024-04-25T09:04:28.400 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:04:28.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/1/new.json 2024-04-25T09:04:28.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbHCpm/3BOORAAwCoXa+BIOnWT+Cr/NNLzUw== --osd-uuid 009a2c57-76f8-4e70-9504-17bdf92177b0 2024-04-25T09:04:28.459 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:28.467+0000 7fb39f6087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:28.492 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:28.503+0000 7fb39f6087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:28.494 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:28.503+0000 7fb39f6087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:28.494 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:28.503+0000 7fb39f6087c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:04:28.494 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:28.503+0000 7fb39f6087c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:04:28.494 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:28.503+0000 7fb39f6087c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:04:28.597 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:28.611+0000 7fb39f6087c0 -1 bluestore(td/pg-split-merge/1) _read_fsid unparsable uuid 2024-04-25T09:04:35.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/1/keyring 2024-04-25T09:04:35.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:04:35.338 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T09:04:35.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:04:35.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:04:35.888 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:04:35.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:04:35.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:04:35.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:04:35.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:04:35.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:04:35.924 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:35.935+0000 7f2ed3c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:35.948 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:35.959+0000 7f2ed3c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:35.949 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:35.959+0000 7f2ed3c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:36.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:04:36.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:04:36.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:04:36.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:04:36.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:04:36.320 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:04:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:04:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:36.513 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:36.523+0000 7f2ed3c807c0 -1 Falling back to public interface 2024-04-25T09:04:36.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:37.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:37.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:37.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:04:37.774 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:04:37.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:37.774 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:38.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:39.006 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:04:39.015+0000 7f2ed3c807c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:04:39.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:39.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:39.184 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:04:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:04:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:39.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:39.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:40.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:40.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:40.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:04:40.648 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:04:40.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:40.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:41.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:42.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:42.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:42.176 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:04:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:04:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:42.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:42.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:43.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:43.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:43.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:04:43.671 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:04:43.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:43.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:44.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:45.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:45.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:45.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:04:45.205 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:04:45.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:45.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:45.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:46.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:46.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:46.663 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:04:46.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:04:46.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:46.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:47.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:48.152 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:04:48.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:04:48.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:48.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:48.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:49.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:49.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:49.592 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:04:49.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:04:49.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:49.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:50.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:51.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:51.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:51.223 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:04:51.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:04:51.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:51.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:51.867 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:52.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:52.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:52.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:04:52.870 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:04:52.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:52.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:54.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:54.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:54.657 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:04:54.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:04:54.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:54.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:55.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:56.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:56.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:56.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:04:56.734 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:04:56.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:56.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:57.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:58.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:58.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:58.647 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:04:58.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:04:58.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:58.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:04:59.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:00.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:00.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:00.291 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:05:00.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:05:00.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:00.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:01.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:02.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:02.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:02.430 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:05:02.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:05:02.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:02.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:03.020 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:04.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:04.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:04.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:05:04.022 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:05:04.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:04.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:05:04.499 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3676604933,v1:127.0.0.1:6811/3676604933] [v2:127.0.0.1:6812/3676604933,v1:127.0.0.1:6813/3676604933] exists,up 009a2c57-76f8-4e70-9504-17bdf92177b0 2024-04-25T09:05:04.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:28: TEST_a_merge_empty: run_osd td/pg-split-merge 2 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/2 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:05:04.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:05:04.504 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/2 2024-04-25T09:05:04.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:05:04.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a443aeb-d712-47ab-afef-272cf876aa9e 2024-04-25T09:05:04.506 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 0a443aeb-d712-47ab-afef-272cf876aa9e 2024-04-25T09:05:04.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0a443aeb-d712-47ab-afef-272cf876aa9e' 2024-04-25T09:05:04.506 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:05:04.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAHCpmmzY8IBAA6wiDrqf/nSZDhXY3wcOVaQ== 2024-04-25T09:05:04.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAHCpmmzY8IBAA6wiDrqf/nSZDhXY3wcOVaQ=="}' 2024-04-25T09:05:04.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a443aeb-d712-47ab-afef-272cf876aa9e -i td/pg-split-merge/2/new.json 2024-04-25T09:05:05.150 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:05:05.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/2/new.json 2024-04-25T09:05:05.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAHCpmmzY8IBAA6wiDrqf/nSZDhXY3wcOVaQ== --osd-uuid 0a443aeb-d712-47ab-afef-272cf876aa9e 2024-04-25T09:05:05.206 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:05.214+0000 7f1142a1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:05.231 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:05.242+0000 7f1142a1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:05.233 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:05.242+0000 7f1142a1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:05.233 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:05.242+0000 7f1142a1b7c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:05:05.233 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:05.242+0000 7f1142a1b7c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:05:05.234 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:05.242+0000 7f1142a1b7c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:05:05.304 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:05.314+0000 7f1142a1b7c0 -1 bluestore(td/pg-split-merge/2) _read_fsid unparsable uuid 2024-04-25T09:05:12.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/2/keyring 2024-04-25T09:05:12.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:05:12.025 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T09:05:12.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:05:12.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:05:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:05:12.712 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:05:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:05:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:05:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:05:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:05:12.742 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:12.750+0000 7fa10ecb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:12.773 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:12.782+0000 7fa10ecb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:12.782+0000 7fa10ecb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:05:13.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:05:13.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:05:13.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:05:13.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:05:13.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:05:13.155 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:05:13.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:13.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:05:13.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:13.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:13.361 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:13.370+0000 7fa10ecb97c0 -1 Falling back to public interface 2024-04-25T09:05:13.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:14.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:14.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:14.608 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:05:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:05:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:15.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:15.952 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:05:15.962+0000 7fa10ecb97c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:05:16.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:16.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:16.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:05:16.015 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:05:16.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:16.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:16.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:17.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:17.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:17.569 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:05:17.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:05:17.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:17.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:18.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:19.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:19.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:19.023 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:05:19.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:05:19.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:19.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:19.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:20.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:20.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:20.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:05:20.435 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:05:20.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:20.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:21.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:22.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:22.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:22.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:05:22.040 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:05:22.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:22.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:22.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:23.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:23.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:23.484 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:05:23.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:05:23.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:23.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:24.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:25.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:05:25.048 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:25.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:25.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:26.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:26.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:26.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:05:26.600 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:05:26.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:26.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:27.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:28.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:28.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:28.137 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:05:28.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:05:28.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:28.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:28.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:29.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:29.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:29.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:05:29.612 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:05:29.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:29.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:30.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:31.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:31.396 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:05:31.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:31.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:05:31.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:31.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:32.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:33.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:33.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:33.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:05:33.405 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:05:33.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:33.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:33.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:34.826 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:05:34.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:34.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:34.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:05:34.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:34.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:35.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:36.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:36.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:36.547 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:05:36.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:05:36.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:36.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:37.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:38.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:38.690 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:05:38.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:38.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:05:38.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:38.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:39.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:40.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:40.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:40.466 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:05:40.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:05:40.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:40.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:41.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:42.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:42.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:42.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:05:42.047 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:05:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:42.048 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:05:42.479 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1984564037,v1:127.0.0.1:6819/1984564037] [v2:127.0.0.1:6820/1984564037,v1:127.0.0.1:6821/1984564037] exists,up 0a443aeb-d712-47ab-afef-272cf876aa9e 2024-04-25T09:05:42.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:05:42.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:05:42.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:05:42.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:30: TEST_a_merge_empty: ceph osd pool create foo 2 2024-04-25T09:05:42.995 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T09:05:43.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:31: TEST_a_merge_empty: ceph osd pool set foo pgp_num 1 2024-04-25T09:05:44.174 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 pgp_num to 1 2024-04-25T09:05:44.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:33: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:05:44.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:05:44.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:05:44.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:05:44.198 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:05:44.199 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:05:44.199 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:05:44.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:05:44.199 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:05:44.200 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:05:44.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:05:44.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:05:44.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:05:44.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:05:44.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:05:44.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:05:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:05:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:05:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:05:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:05:44.951 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:44.951 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:05:45.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-04-25T09:05:45.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-04-25T09:05:45.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-04-25T09:05:45.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:45.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:05:45.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T09:05:45.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T09:05:45.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672978' 2024-04-25T09:05:45.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:45.306 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:05:45.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-04-25T09:05:45.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-04-25T09:05:45.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-64424509451' 2024-04-25T09:05:45.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:45.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-04-25T09:05:45.458 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:45.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:05:45.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-04-25T09:05:45.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:45.461 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836500 2024-04-25T09:05:45.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-04-25T09:05:45.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-25T09:05:45.461 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:45.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836500 2024-04-25T09:05:45.879 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:45.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T09:05:45.880 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:45.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:05:45.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T09:05:45.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:45.883 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672978 2024-04-25T09:05:45.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T09:05:45.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T09:05:45.884 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:05:46.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-04-25T09:05:46.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:46.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-04-25T09:05:46.350 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:46.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:05:46.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-04-25T09:05:46.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:46.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-04-25T09:05:46.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-25T09:05:46.354 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509451 2024-04-25T09:05:46.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:05:46.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509451 2024-04-25T09:05:46.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:05:47.767 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:05:47.767 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:05:48.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509451 2024-04-25T09:05:48.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:05:48.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:48.265 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:48.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:05:48.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:48.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:48.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:48.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:48.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:48.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:48.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:49.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:05:49.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:49.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:49.372 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:49.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:05:49.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:05:49.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:05:49.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:37: TEST_a_merge_empty: rados -p foo put foo1 /etc/passwd 2024-04-25T09:05:50.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:38: TEST_a_merge_empty: rados -p foo put foo2 /etc/passwd 2024-04-25T09:05:50.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:39: TEST_a_merge_empty: rados -p foo put foo3 /etc/passwd 2024-04-25T09:05:50.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:40: TEST_a_merge_empty: rados -p foo put foo4 /etc/passwd 2024-04-25T09:05:50.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:42: TEST_a_merge_empty: rados -p foo put foo5 /etc/passwd 2024-04-25T09:05:50.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:43: TEST_a_merge_empty: rados -p foo put foo6 /etc/passwd 2024-04-25T09:05:50.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:44: TEST_a_merge_empty: rados -p foo put foo8 /etc/passwd 2024-04-25T09:05:51.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:45: TEST_a_merge_empty: rados -p foo put foo10 /etc/passwd 2024-04-25T09:05:51.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:46: TEST_a_merge_empty: rados -p foo put foo11 /etc/passwd 2024-04-25T09:05:51.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:47: TEST_a_merge_empty: rados -p foo put foo12 /etc/passwd 2024-04-25T09:05:51.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:48: TEST_a_merge_empty: rados -p foo put foo16 /etc/passwd 2024-04-25T09:05:51.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:50: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:05:51.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:05:51.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:05:51.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:05:51.781 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:05:51.781 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:05:51.781 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:05:51.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:05:51.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:05:51.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:05:51.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:05:51.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:05:51.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:05:51.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:05:51.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:05:51.892 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:05:52.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:05:52.358 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:05:52.358 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:05:52.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:05:52.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:52.358 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:05:52.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-04-25T09:05:52.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-04-25T09:05:52.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-04-25T09:05:52.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:52.552 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:05:52.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-04-25T09:05:52.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-04-25T09:05:52.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672981' 2024-04-25T09:05:52.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:52.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:05:52.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-04-25T09:05:52.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-04-25T09:05:52.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672981 2-64424509455' 2024-04-25T09:05:52.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:52.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-04-25T09:05:52.871 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:52.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:05:52.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-04-25T09:05:52.873 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:52.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-04-25T09:05:52.874 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836503 2024-04-25T09:05:52.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-25T09:05:52.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836503 2024-04-25T09:05:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:05:54.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:05:54.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:54.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836503 2024-04-25T09:05:54.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:54.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-04-25T09:05:54.713 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:54.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:05:54.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-04-25T09:05:54.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:54.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-04-25T09:05:54.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-04-25T09:05:54.716 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672981 2024-04-25T09:05:54.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:05:55.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672981 -lt 42949672981 2024-04-25T09:05:55.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:55.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:55.168 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-04-25T09:05:55.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:05:55.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-04-25T09:05:55.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:55.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-04-25T09:05:55.171 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509455 2024-04-25T09:05:55.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-25T09:05:55.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:05:55.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-04-25T09:05:55.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:05:55.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:55.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:56.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:05:56.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:56.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:56.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:56.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:56.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:56.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:56.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:56.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:05:56.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:56.547 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:56.548 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:05:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:05:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:05:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:52: TEST_a_merge_empty: ceph tell osd.1 config set osd_debug_no_purge_strays true 2024-04-25T09:05:57.245 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T09:05:57.246 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "" 2024-04-25T09:05:57.246 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T09:05:57.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:53: TEST_a_merge_empty: ceph osd pool set foo size 2 2024-04-25T09:05:58.214 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 2 2024-04-25T09:05:58.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:54: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:05:58.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:05:58.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:05:58.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:05:58.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:05:58.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:05:58.240 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:05:58.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:05:58.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:05:58.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:05:58.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:05:58.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:05:58.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:05:58.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:05:58.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:05:58.346 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:05:58.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:05:58.864 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:05:58.864 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:05:58.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:05:58.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:58.864 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:05:59.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-04-25T09:05:59.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-04-25T09:05:59.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-04-25T09:05:59.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:59.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:05:59.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672984 2024-04-25T09:05:59.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672984 2024-04-25T09:05:59.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672984' 2024-04-25T09:05:59.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:59.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:05:59.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-04-25T09:05:59.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-04-25T09:05:59.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672984 2-64424509458' 2024-04-25T09:05:59.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:59.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-04-25T09:05:59.332 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:59.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:05:59.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:59.333 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-04-25T09:05:59.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-04-25T09:05:59.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-04-25T09:05:59.335 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836507 2024-04-25T09:05:59.335 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:59.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836507 2024-04-25T09:05:59.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:59.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672984 2024-04-25T09:05:59.721 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:59.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:05:59.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672984 2024-04-25T09:05:59.724 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:59.726 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672984 2024-04-25T09:05:59.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672984 2024-04-25T09:05:59.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-04-25T09:05:59.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:06:00.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672984 -lt 42949672984 2024-04-25T09:06:00.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:00.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-04-25T09:06:00.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:00.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:06:00.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-04-25T09:06:00.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:00.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-04-25T09:06:00.177 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509458 2024-04-25T09:06:00.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-25T09:06:00.177 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:06:00.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509458 2024-04-25T09:06:00.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:06:01.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:06:01.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:06:02.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-04-25T09:06:02.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:06:02.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:02.091 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:02.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:06:02.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:02.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:02.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:02.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:02.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:02.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:02.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:03.054 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:06:03.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:03.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:03.055 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:03.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:06:03.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:06:03.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:06:03.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:56: TEST_a_merge_empty: kill_daemons td/pg-split-merge TERM osd.2 2024-04-25T09:06:03.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:06:03.600 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:06:03.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:06:03.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:06:03.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:06:03.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:06:03.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:57: TEST_a_merge_empty: ceph-objectstore-tool --data-path td/pg-split-merge/2 --op remove --pgid 1.1 --force 2024-04-25T09:06:05.177 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T09:06:05.178 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T09:06:05.178 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:06:05.302 INFO:tasks.workunit.client.0.smithi088.stdout:Remove successful 2024-04-25T09:06:05.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:58: TEST_a_merge_empty: activate_osd td/pg-split-merge 2 2024-04-25T09:06:05.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:06:05.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:06:05.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/2 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:06:05.716 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:06:05.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:06:05.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:06:05.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:05.717 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:05.717 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:06:05.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:06:05.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:06:05.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:06:05.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:06:05.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:06:05.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:06:05.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:06:05.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:06:05.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:06:05.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:06:05.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/2 2024-04-25T09:06:05.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T09:06:05.719 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:06:05.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:06:05.720 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/2/whoami 2024-04-25T09:06:05.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T09:06:05.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:06:05.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:06:05.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:06:05.753 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:05.765+0000 7f3c8292a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:05.756 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:05.769+0000 7f3c8292a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:05.758 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:05.769+0000 7f3c8292a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:06.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T09:06:06.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:06:06.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:06:06.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:06:06.167 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:06:06.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:06:06.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:06.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:06:06.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:06.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:06:06.336 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:06.349+0000 7f3c8292a7c0 -1 Falling back to public interface 2024-04-25T09:06:06.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:07.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:07.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:07.646 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:06:07.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:06:07.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:07.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:06:08.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:09.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:09.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:09.099 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:06:09.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:06:09.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:09.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:06:09.137 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:09.149+0000 7f3c8292a7c0 -1 osd.2 28 log_to_monitors true 2024-04-25T09:06:09.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:10.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:10.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:10.532 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:06:10.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:06:10.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:10.532 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:06:10.939 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 31 up_thru 17 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/3302127554,v1:127.0.0.1:6819/3302127554] [v2:127.0.0.1:6820/3302127554,v1:127.0.0.1:6821/3302127554] exists,up 0a443aeb-d712-47ab-afef-272cf876aa9e 2024-04-25T09:06:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:06:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:06:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:06:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:60: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:06:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:06:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:06:10.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:06:10.940 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:06:10.941 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:06:10.941 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:06:10.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:06:10.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:06:10.941 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:06:11.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:06:11.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:06:11.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:06:11.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:06:11.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:06:11.041 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:06:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:06:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:06:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:06:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:06:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:11.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:06:11.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T09:06:11.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T09:06:11.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T09:06:11.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:11.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:06:11.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-04-25T09:06:11.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-04-25T09:06:11.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672989' 2024-04-25T09:06:11.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:11.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:06:11.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=133143986179 2024-04-25T09:06:11.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 133143986179 2024-04-25T09:06:11.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672989 2-133143986179' 2024-04-25T09:06:11.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:11.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T09:06:11.872 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:11.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:06:11.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T09:06:11.874 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:11.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T09:06:11.875 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836511 2024-04-25T09:06:11.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T09:06:11.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:06:12.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836511 2024-04-25T09:06:12.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:06:13.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:06:13.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:06:13.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836511 2024-04-25T09:06:13.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:13.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-04-25T09:06:13.722 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:13.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:06:13.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-04-25T09:06:13.725 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:13.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-04-25T09:06:13.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-04-25T09:06:13.726 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672989 2024-04-25T09:06:13.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:06:14.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-04-25T09:06:14.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:14.189 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-133143986179 2024-04-25T09:06:14.190 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:14.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:06:14.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-133143986179 2024-04-25T09:06:14.191 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:14.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=133143986179 2024-04-25T09:06:14.193 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 133143986179 2024-04-25T09:06:14.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 133143986179' 2024-04-25T09:06:14.193 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:06:14.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 133143986179 -lt 133143986179 2024-04-25T09:06:14.598 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:06:14.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:14.599 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:15.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:06:15.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:15.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:15.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:15.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:15.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:15.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:15.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:15.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:06:15.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:15.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:15.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:16.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:06:16.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:06:16.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:06:16.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:66: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change true 2024-04-25T09:06:16.287 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T09:06:16.287 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "" 2024-04-25T09:06:16.287 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T09:06:16.301 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:67: TEST_a_merge_empty: ceph osd out 0 1 2024-04-25T09:06:16.830 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already out. osd.1 is already out. 2024-04-25T09:06:16.846 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:68: TEST_a_merge_empty: ceph osd pool set foo pg_num 1 2024-04-25T09:06:17.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:69: TEST_a_merge_empty: sleep 5 2024-04-25T09:06:22.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:70: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change false 2024-04-25T09:06:22.981 INFO:tasks.workunit.client.0.smithi088.stdout:{ 2024-04-25T09:06:22.981 INFO:tasks.workunit.client.0.smithi088.stdout: "success": "" 2024-04-25T09:06:22.981 INFO:tasks.workunit.client.0.smithi088.stdout:} 2024-04-25T09:06:22.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:74: TEST_a_merge_empty: ceph osd in 0 1 2024-04-25T09:06:23.525 INFO:tasks.workunit.client.0.smithi088.stderr:osd.0 is already in. osd.1 is already in. 2024-04-25T09:06:23.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:75: TEST_a_merge_empty: ceph osd pool set foo size 3 2024-04-25T09:06:24.860 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 3 2024-04-25T09:06:24.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:77: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:06:24.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:06:24.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:06:24.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:06:24.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:06:24.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:06:24.881 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:06:24.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:06:24.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:06:24.882 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:06:24.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:06:24.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:06:24.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:06:24.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:06:24.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:06:24.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:06:25.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:06:25.555 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:06:25.555 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:06:25.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:06:25.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:25.555 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:06:25.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-04-25T09:06:25.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-04-25T09:06:25.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-04-25T09:06:25.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:25.715 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:06:26.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672994 2024-04-25T09:06:26.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672994 2024-04-25T09:06:26.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672994' 2024-04-25T09:06:26.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:06:26.017 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:06:26.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=133143986184 2024-04-25T09:06:26.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 133143986184 2024-04-25T09:06:26.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672994 2-133143986184' 2024-04-25T09:06:26.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:26.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-04-25T09:06:26.179 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:26.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:06:26.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-04-25T09:06:26.180 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:26.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-04-25T09:06:26.181 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836516 2024-04-25T09:06:26.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-04-25T09:06:26.182 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:06:26.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836514 -lt 21474836516 2024-04-25T09:06:26.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:06:27.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:06:27.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:06:28.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836516 -lt 21474836516 2024-04-25T09:06:28.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:28.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:28.169 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672994 2024-04-25T09:06:28.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:06:28.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672994 2024-04-25T09:06:28.170 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:28.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672994 2024-04-25T09:06:28.172 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672994 2024-04-25T09:06:28.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-04-25T09:06:28.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:06:28.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672994 -lt 42949672994 2024-04-25T09:06:28.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:06:28.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:06:28.573 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-133143986184 2024-04-25T09:06:28.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:06:28.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-133143986184 2024-04-25T09:06:28.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:06:28.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=133143986184 2024-04-25T09:06:28.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 133143986184' 2024-04-25T09:06:28.576 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 133143986184 2024-04-25T09:06:28.576 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:06:28.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 133143986184 -lt 133143986184 2024-04-25T09:06:28.990 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:06:28.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:28.991 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:29.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:06:29.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:29.580 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:29.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:29.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:29.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:29.992 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:30.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:30.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:06:30.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:06:30.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:06:30.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:30.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:30.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:30.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:30.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:30.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:30.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:30.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:30.623 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:30.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:31.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:31.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:31.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:31.018 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:31.594 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:31.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:31.595 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:32.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8401 2024-04-25T09:06:32.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8401 '!=' null 2024-04-25T09:06:32.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:32.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:32.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:32.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:32.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:32.259 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:32.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:06:32.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:32.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:32.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:06:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:06:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:06:33.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:81: TEST_a_merge_empty: ceph pg scrub 1.0 2024-04-25T09:06:33.668 INFO:tasks.workunit.client.0.smithi088.stderr:instructing pg 1.0 on osd.1 to scrub 2024-04-25T09:06:33.683 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:82: TEST_a_merge_empty: sleep 10 2024-04-25T09:06:43.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:83: TEST_a_merge_empty: ceph log last debug 2024-04-25T09:06:44.219 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:24.478017+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e40: 3 total, 3 up, 3 in 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:24.943351+0000 mon.a (mon.0) 417 : cluster [DBG] osdmap e41: 3 total, 3 up, 3 in 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:25.355888+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e42: 3 total, 3 up, 3 in 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:25.323231+0000 mgr.x (mgr.4100) 74 : cluster [DBG] pgmap v106: 1 pgs: 1 clean+premerge+peered; 82 KiB data, 54 MiB used, 200 GiB / 200 GiB avail 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:25.581448+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e43: 3 total, 3 up, 3 in 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:26.025354+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e44: 3 total, 3 up, 3 in 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:26.156564+0000 osd.1 (osd.1) 3 : cluster [INF] 1.0 continuing backfill to osd.2 from (25'7,25'7] MIN to 25'7 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:27.073534+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e45: 3 total, 3 up, 3 in 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:27.567756+0000 mon.a (mon.0) 426 : cluster [DBG] osdmap e46: 3 total, 3 up, 3 in 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:27.323783+0000 mgr.x (mgr.4100) 75 : cluster [DBG] pgmap v111: 1 pgs: 1 remapped+peering; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 20 KiB/s wr, 1 op/s 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:29.324292+0000 mgr.x (mgr.4100) 76 : cluster [DBG] pgmap v113: 1 pgs: 1 remapped+peering; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 12 KiB/s wr, 1 op/s 2024-04-25T09:06:44.220 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:31.324826+0000 mgr.x (mgr.4100) 81 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 8.4 KiB/s wr, 0 op/s; 8.2 KiB/s, 0 objects/s recovering 2024-04-25T09:06:44.221 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:33.325327+0000 mgr.x (mgr.4100) 84 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 6.5 KiB/s, 0 objects/s recovering 2024-04-25T09:06:44.221 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:34.247403+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub starts 2024-04-25T09:06:44.221 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:34.252624+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub ok 2024-04-25T09:06:44.221 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:35.325889+0000 mgr.x (mgr.4100) 87 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 5.7 KiB/s, 0 objects/s recovering 2024-04-25T09:06:44.221 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:37.326406+0000 mgr.x (mgr.4100) 88 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.7 KiB/s, 0 objects/s recovering 2024-04-25T09:06:44.221 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:39.326943+0000 mgr.x (mgr.4100) 89 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.0 KiB/s, 0 objects/s recovering 2024-04-25T09:06:44.221 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:41.327478+0000 mgr.x (mgr.4100) 90 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 3.9 KiB/s, 0 objects/s recovering 2024-04-25T09:06:44.221 INFO:tasks.workunit.client.0.smithi088.stdout:2024-04-25T09:06:43.327987+0000 mgr.x (mgr.4100) 91 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail 2024-04-25T09:06:44.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: ceph pg ls 2024-04-25T09:06:44.649 INFO:tasks.workunit.client.0.smithi088.stdout:PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 2024-04-25T09:06:44.650 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 11 0 0 0 132748 0 0 0 11 active+clean 10s 25'7 46:100 [1,0,2]p1 [1,0,2]p1 2024-04-25T09:06:34.252637+0000 2024-04-25T09:05:42.854537+0000 1 periodic scrub scheduled @ 2024-04-26T18:32:37.537495+0000 2024-04-25T09:06:44.650 INFO:tasks.workunit.client.0.smithi088.stdout: 2024-04-25T09:06:44.650 INFO:tasks.workunit.client.0.smithi088.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:06:44.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: ceph pg ls 2024-04-25T09:06:44.665 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: grep ' active.clean ' 2024-04-25T09:06:45.079 INFO:tasks.workunit.client.0.smithi088.stdout:1.0 11 0 0 0 132748 0 0 0 11 active+clean 10s 25'7 46:100 [1,0,2]p1 [1,0,2]p1 2024-04-25T09:06:34.252637+0000 2024-04-25T09:05:42.854537+0000 1 periodic scrub scheduled @ 2024-04-26T18:32:37.537495+0000 2024-04-25T09:06:45.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:06:45.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:06:45.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:06:45.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:06:45.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:06:45.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:06:45.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:06:45.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:06:45.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:06:45.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:06:45.392 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:06:45.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:06:45.395 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:06:45.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:06:45.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:06:45.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:06:45.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:06:45.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:06:45.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:06:45.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:06:45.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:06:45.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:06:45.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:06:45.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:06:45.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:45.423 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:45.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451260 2024-04-25T09:06:45.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:06:45.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:06:45.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:06:45.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:06:45.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:06:45.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:06:45.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:06:45.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:06:45.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:06:45.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:06:45.426 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:06:45.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:06:45.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:06:45.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:06:45.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:06:45.429 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:06:45.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:06:45.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:06:45.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:06:45.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:06:45.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:06:45.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:06:45.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:06:45.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:06:45.435 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:06:45.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:06:45.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:06:45.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:06:45.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:06:45.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:45.438 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451260 2024-04-25T09:06:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:06:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:06:45.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:06:45.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:06:45.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:45.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:45.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.451260 2024-04-25T09:06:45.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:06:45.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:06:45.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:06:45.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:06:45.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_merge_and_gap td/pg-split-merge 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:89: TEST_import_after_merge_and_gap: local dir=td/pg-split-merge 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:91: TEST_import_after_merge_and_gap: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:06:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:06:45.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:06:45.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:06:45.925 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:06:45.926 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:06:45.926 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:45.926 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:45.926 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:06:45.926 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:06:46.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:06:46.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:06:46.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:06:46.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:06:46.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:06:46.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:06:46.139 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:06:46.139 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:06:46.139 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:06:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:06:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:46.140 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451260/ceph-mon.a.asok 2024-04-25T09:06:46.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:06:46.140 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451260/ceph-mon.a.asok config get fsid 2024-04-25T09:06:46.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:06:46.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:06:46.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:06:46.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:06:46.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:06:46.249 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:06:46.249 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:06:46.249 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:06:46.249 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:06:46.249 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:46.250 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:46.250 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451260/ceph-mon.a.asok 2024-04-25T09:06:46.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:06:46.250 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451260/ceph-mon.a.asok config get mon_host 2024-04-25T09:06:46.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:92: TEST_import_after_merge_and_gap: run_mgr td/pg-split-merge x 2024-04-25T09:06:46.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:06:46.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:06:46.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:06:46.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:06:46.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:06:46.349 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:06:47.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:06:47.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:06:47.256 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:06:47.257 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:06:47.257 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:47.257 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:47.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:06:47.257 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:06:47.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:93: TEST_import_after_merge_and_gap: run_osd td/pg-split-merge 0 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:06:47.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:06:47.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:06:47.297 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:06:47.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=168d882f-4939-4337-8b16-26a6cead1b6d 2024-04-25T09:06:47.298 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 168d882f-4939-4337-8b16-26a6cead1b6d 2024-04-25T09:06:47.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 168d882f-4939-4337-8b16-26a6cead1b6d' 2024-04-25T09:06:47.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:06:47.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnHSpmQKbMExAATkwdbRASURdBGWRTvceJgw== 2024-04-25T09:06:47.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnHSpmQKbMExAATkwdbRASURdBGWRTvceJgw=="}' 2024-04-25T09:06:47.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 168d882f-4939-4337-8b16-26a6cead1b6d -i td/pg-split-merge/0/new.json 2024-04-25T09:06:47.640 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:06:47.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:06:47.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnHSpmQKbMExAATkwdbRASURdBGWRTvceJgw== --osd-uuid 168d882f-4939-4337-8b16-26a6cead1b6d 2024-04-25T09:06:47.692 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:47.704+0000 7fea32a3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:47.717 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:47.728+0000 7fea32a3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:47.718 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:47.728+0000 7fea32a3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:47.718 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:47.728+0000 7fea32a3f7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:06:47.719 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:47.728+0000 7fea32a3f7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:06:47.719 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:47.728+0000 7fea32a3f7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:06:47.757 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:47.768+0000 7fea32a3f7c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:06:54.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:06:54.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:06:54.077 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:06:54.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:06:54.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:06:54.634 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:06:54.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:06:54.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:06:54.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:06:54.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:06:54.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:06:54.667 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:54.680+0000 7f9604d5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:54.689 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:54.700+0000 7f9604d5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:54.690 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:54.700+0000 7f9604d5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:55.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:06:55.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:06:55.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:06:55.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:06:55.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:06:55.042 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:06:55.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:55.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:06:55.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:55.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:55.249 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:55.260+0000 7f9604d5a7c0 -1 Falling back to public interface 2024-04-25T09:06:55.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:56.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:56.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:56.545 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:06:56.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:06:56.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:56.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:56.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:57.832 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:06:57.844+0000 7f9604d5a7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:06:57.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:57.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:57.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:06:57.983 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:06:57.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:57.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:58.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:59.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:59.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:59.514 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:06:59.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:06:59.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:59.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:00.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:01.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:01.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:01.065 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:07:01.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:07:01.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:01.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:01.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:02.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:02.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:02.495 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:07:02.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:07:02.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:02.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:02.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:04.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:04.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:04.001 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:07:04.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:07:04.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:04.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:04.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:05.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:05.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:05.465 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:07:05.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:07:05.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:05.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:06.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:07.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:07.031 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:07.031 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:07:07.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:07:07.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:07.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:07.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:08.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:08.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:08.483 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:07:08.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:07:08.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:08.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:09.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:10.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:10.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:10.008 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:07:10.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:07:10.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:10.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:10.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:11.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:11.429 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:07:11.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:07:11.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:11.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:12.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:13.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:13.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:13.039 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:07:13.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:07:13.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:13.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:13.856 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:14.857 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:14.858 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:14.858 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:07:14.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:07:14.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:14.859 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:15.519 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:16.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:16.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:16.521 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:07:16.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:07:16.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:16.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:17.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:18.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:18.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:18.496 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:07:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:07:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:19.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:20.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:20.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:20.449 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:07:20.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:07:20.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:20.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:21.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:22.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:22.486 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:07:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:07:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:22.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:22.909 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1571925411,v1:127.0.0.1:6803/1571925411] [v2:127.0.0.1:6804/1571925411,v1:127.0.0.1:6805/1571925411] exists,up 168d882f-4939-4337-8b16-26a6cead1b6d 2024-04-25T09:07:22.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:07:22.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:07:22.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:07:22.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:95: TEST_import_after_merge_and_gap: ceph osd pool create foo 2 2024-04-25T09:07:23.463 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T09:07:23.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:96: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:07:23.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:07:23.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:07:23.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:07:23.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:07:23.480 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:07:23.481 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:07:23.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:07:23.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:07:23.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:07:23.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:07:23.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:07:23.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:07:23.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:07:23.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:07:23.585 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:07:24.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:07:24.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:07:24.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:24.154 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:07:24.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836486 2024-04-25T09:07:24.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836486 2024-04-25T09:07:24.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836486' 2024-04-25T09:07:24.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:24.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836486 2024-04-25T09:07:24.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:24.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:07:24.404 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836486 2024-04-25T09:07:24.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:24.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836486 2024-04-25T09:07:24.406 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836486 2024-04-25T09:07:24.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-04-25T09:07:24.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:07:24.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836484 -lt 21474836486 2024-04-25T09:07:24.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:07:25.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:07:25.810 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:07:26.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836486 -lt 21474836486 2024-04-25T09:07:26.226 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:07:26.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:07:26.227 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:07:26.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:07:26.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:07:26.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:07:26.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:07:26.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:07:26.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:07:26.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:07:26.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:07:27.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:07:27.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:07:27.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:07:27.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:07:27.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:07:27.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:07:27.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:07:27.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:97: TEST_import_after_merge_and_gap: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-04-25T09:07:27.859 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T09:07:27.859 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-04-25T09:07:27.859 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_462306 2024-04-25T09:07:27.859 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:07:27.859 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:07:28.859 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 114 98 0.0956911 0.0957031 0.125294 0.147983 2024-04-25T09:07:29.859 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 240 224 0.10936 0.123047 0.142856 0.140206 2024-04-25T09:07:30.859 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 338 322 0.104805 0.0957031 0.167437 0.14348 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 3.10872 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 338 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 1024 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 1024 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.106178 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.0157869 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.123047 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.0957031 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 108 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 16.1658 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 126 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 98 2024-04-25T09:07:31.860 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.14439 2024-04-25T09:07:31.861 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0298602 2024-04-25T09:07:31.861 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.243211 2024-04-25T09:07:31.861 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.0937673 2024-04-25T09:07:31.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:99: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:07:31.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:07:31.974 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:07:31.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:07:31.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:07:31.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:07:32.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:07:32.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:100: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-04-25T09:07:32.485 INFO:tasks.workunit.client.0.smithi088.stderr:Exporting 1.1 info 1.1( v 8'181 (0'0,8'181] local-lis/les=6/7 n=181 ec=6/6 lis/c=6/6 les/c/f=7/7/0 sis=6) 2024-04-25T09:07:32.489 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81099aed:::benchmark_data_smithi088_462306_object181:head# 2024-04-25T09:07:32.491 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81ae27f9:::benchmark_data_smithi088_462306_object252:head# 2024-04-25T09:07:32.493 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81b12e8e:::benchmark_data_smithi088_462306_object161:head# 2024-04-25T09:07:32.494 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81e3c822:::benchmark_data_smithi088_462306_object64:head# 2024-04-25T09:07:32.494 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:82743540:::benchmark_data_smithi088_462306_object276:head# 2024-04-25T09:07:32.495 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:842cb2a9:::benchmark_data_smithi088_462306_object278:head# 2024-04-25T09:07:32.495 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:84c54114:::benchmark_data_smithi088_462306_object66:head# 2024-04-25T09:07:32.496 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:85078920:::benchmark_data_smithi088_462306_object62:head# 2024-04-25T09:07:32.497 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:85f7c2db:::benchmark_data_smithi088_462306_object326:head# 2024-04-25T09:07:32.497 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8770f696:::benchmark_data_smithi088_462306_object253:head# 2024-04-25T09:07:32.497 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87b0211e:::benchmark_data_smithi088_462306_object8:head# 2024-04-25T09:07:32.513 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87cd1abd:::benchmark_data_smithi088_462306_object128:head# 2024-04-25T09:07:32.515 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87cf274d:::benchmark_data_smithi088_462306_object19:head# 2024-04-25T09:07:32.516 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87f67271:::benchmark_data_smithi088_462306_object189:head# 2024-04-25T09:07:32.517 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:88f3d2ff:::benchmark_data_smithi088_462306_object173:head# 2024-04-25T09:07:32.517 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8a4612ce:::benchmark_data_smithi088_462306_object233:head# 2024-04-25T09:07:32.518 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8bb6f309:::benchmark_data_smithi088_462306_object132:head# 2024-04-25T09:07:32.519 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8ca22c61:::benchmark_data_smithi088_462306_object220:head# 2024-04-25T09:07:32.519 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8d801fc0:::benchmark_data_smithi088_462306_object207:head# 2024-04-25T09:07:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8fbb559d:::benchmark_data_smithi088_462306_object257:head# 2024-04-25T09:07:32.520 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:91623a25:::benchmark_data_smithi088_462306_object138:head# 2024-04-25T09:07:32.521 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9196f898:::benchmark_data_smithi088_462306_object6:head# 2024-04-25T09:07:32.521 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:91e42837:::benchmark_data_smithi088_462306_object176:head# 2024-04-25T09:07:32.522 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:91e98fd5:::benchmark_data_smithi088_462306_object35:head# 2024-04-25T09:07:32.522 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:92220cfe:::benchmark_data_smithi088_462306_object22:head# 2024-04-25T09:07:32.523 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9265404e:::benchmark_data_smithi088_462306_object169:head# 2024-04-25T09:07:32.523 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:92ad4e27:::benchmark_data_smithi088_462306_object238:head# 2024-04-25T09:07:32.523 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:934c91ea:::benchmark_data_smithi088_462306_object246:head# 2024-04-25T09:07:32.524 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:937e9a49:::benchmark_data_smithi088_462306_object199:head# 2024-04-25T09:07:32.524 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:93b8c46b:::benchmark_data_smithi088_462306_object76:head# 2024-04-25T09:07:32.524 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9411c558:::benchmark_data_smithi088_462306_object24:head# 2024-04-25T09:07:32.525 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:94abe535:::benchmark_data_smithi088_462306_object60:head# 2024-04-25T09:07:32.525 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:96cf9e3a:::benchmark_data_smithi088_462306_object325:head# 2024-04-25T09:07:32.525 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:98888887:::benchmark_data_smithi088_462306_object130:head# 2024-04-25T09:07:32.525 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:98cc271a:::benchmark_data_smithi088_462306_object174:head# 2024-04-25T09:07:32.526 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9935b1ed:::benchmark_data_smithi088_462306_object272:head# 2024-04-25T09:07:32.526 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9938cea0:::benchmark_data_smithi088_462306_object162:head# 2024-04-25T09:07:32.527 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9acb5316:::benchmark_data_smithi088_462306_object222:head# 2024-04-25T09:07:32.527 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9b089756:::benchmark_data_smithi088_462306_object271:head# 2024-04-25T09:07:32.527 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9baeb73f:::benchmark_data_smithi088_462306_object117:head# 2024-04-25T09:07:32.528 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9bf57a33:::benchmark_data_smithi088_462306_object204:head# 2024-04-25T09:07:32.528 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9c595d90:::benchmark_data_smithi088_462306_object16:head# 2024-04-25T09:07:32.529 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9caedf93:::benchmark_data_smithi088_462306_object109:head# 2024-04-25T09:07:32.530 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9cb158b2:::benchmark_data_smithi088_462306_object150:head# 2024-04-25T09:07:32.531 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9f6e371a:::benchmark_data_smithi088_462306_object275:head# 2024-04-25T09:07:32.532 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a20735fd:::benchmark_data_smithi088_462306_object282:head# 2024-04-25T09:07:32.532 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a2c8e7c0:::benchmark_data_smithi088_462306_object116:head# 2024-04-25T09:07:32.533 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a3548fec:::benchmark_data_smithi088_462306_object178:head# 2024-04-25T09:07:32.534 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a3d97e4f:::benchmark_data_smithi088_462306_object37:head# 2024-04-25T09:07:32.534 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a40b440b:::benchmark_data_smithi088_462306_object197:head# 2024-04-25T09:07:32.535 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a4fe7d56:::benchmark_data_smithi088_462306_object92:head# 2024-04-25T09:07:32.536 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6c17891:::benchmark_data_smithi088_462306_object146:head# 2024-04-25T09:07:32.537 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6cdc671:::benchmark_data_smithi088_462306_object195:head# 2024-04-25T09:07:32.538 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6ebe5bf:::benchmark_data_smithi088_462306_object145:head# 2024-04-25T09:07:32.538 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a7506a6e:::benchmark_data_smithi088_462306_object122:head# 2024-04-25T09:07:32.539 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a87a78a1:::benchmark_data_smithi088_462306_object5:head# 2024-04-25T09:07:32.539 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a87bb1e0:::benchmark_data_smithi088_462306_object263:head# 2024-04-25T09:07:32.540 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a8de3fc3:::benchmark_data_smithi088_462306_object134:head# 2024-04-25T09:07:32.541 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a92036b2:::benchmark_data_smithi088_462306_object225:head# 2024-04-25T09:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:aa84cf26:::benchmark_data_smithi088_462306_object11:head# 2024-04-25T09:07:32.542 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ab034bbd:::benchmark_data_smithi088_462306_object232:head# 2024-04-25T09:07:32.543 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ab565d7b:::benchmark_data_smithi088_462306_object302:head# 2024-04-25T09:07:32.543 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:abc81ff6:::benchmark_data_smithi088_462306_object69:head# 2024-04-25T09:07:32.544 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ac08c85b:::benchmark_data_smithi088_462306_object61:head# 2024-04-25T09:07:32.545 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:acecab62:::benchmark_data_smithi088_462306_object219:head# 2024-04-25T09:07:32.545 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ad48ca91:::benchmark_data_smithi088_462306_object209:head# 2024-04-25T09:07:32.546 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:af94d379:::benchmark_data_smithi088_462306_object298:head# 2024-04-25T09:07:32.547 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b06befdd:::benchmark_data_smithi088_462306_object156:head# 2024-04-25T09:07:32.547 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b0c2cf35:::benchmark_data_smithi088_462306_object240:head# 2024-04-25T09:07:32.548 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b17d8fef:::benchmark_data_smithi088_462306_object13:head# 2024-04-25T09:07:32.549 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b20bd2ac:::benchmark_data_smithi088_462306_object26:head# 2024-04-25T09:07:32.549 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b2160c47:::benchmark_data_smithi088_462306_object159:head# 2024-04-25T09:07:32.550 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b2f527bb:::benchmark_data_smithi088_462306_object21:head# 2024-04-25T09:07:32.551 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b38816db:::benchmark_data_smithi088_462306_object87:head# 2024-04-25T09:07:32.551 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b39555ce:::benchmark_data_smithi088_462306_object311:head# 2024-04-25T09:07:32.552 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b3d4574c:::benchmark_data_smithi088_462306_object182:head# 2024-04-25T09:07:32.553 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b50b3477:::benchmark_data_smithi088_462306_object55:head# 2024-04-25T09:07:32.554 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b7cd0d37:::benchmark_data_smithi088_462306_object294:head# 2024-04-25T09:07:32.554 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b82e2947:::benchmark_data_smithi088_462306_object163:head# 2024-04-25T09:07:32.555 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:07:32.556 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b8db97a9:::benchmark_data_smithi088_462306_object165:head# 2024-04-25T09:07:32.556 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b97bb4a3:::benchmark_data_smithi088_462306_object319:head# 2024-04-25T09:07:32.557 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba2c6dac:::benchmark_data_smithi088_462306_object251:head# 2024-04-25T09:07:32.557 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba685cfd:::benchmark_data_smithi088_462306_object99:head# 2024-04-25T09:07:32.559 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba7c5b40:::benchmark_data_smithi088_462306_object202:head# 2024-04-25T09:07:32.559 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba8d78f2:::benchmark_data_smithi088_462306_object148:head# 2024-04-25T09:07:32.559 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:baf95fb7:::benchmark_data_smithi088_462306_object290:head# 2024-04-25T09:07:32.560 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bca58a03:::benchmark_data_smithi088_462306_object129:head# 2024-04-25T09:07:32.560 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bdd50353:::benchmark_data_smithi088_462306_object98:head# 2024-04-25T09:07:32.562 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:be02a273:::benchmark_data_smithi088_462306_object291:head# 2024-04-25T09:07:32.562 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bfa07270:::benchmark_data_smithi088_462306_object258:head# 2024-04-25T09:07:32.562 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bfc2ddbb:::benchmark_data_smithi088_462306_object155:head# 2024-04-25T09:07:32.562 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c18f632f:::benchmark_data_smithi088_462306_object43:head# 2024-04-25T09:07:32.563 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c2f3461e:::benchmark_data_smithi088_462306_object91:head# 2024-04-25T09:07:32.564 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c40c5699:::benchmark_data_smithi088_462306_object280:head# 2024-04-25T09:07:32.565 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c4517802:::benchmark_data_smithi088_462306_object3:head# 2024-04-25T09:07:32.565 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c4dec443:::benchmark_data_smithi088_462306_object277:head# 2024-04-25T09:07:32.566 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c588688f:::benchmark_data_smithi088_462306_object273:head# 2024-04-25T09:07:32.567 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c621a2c5:::benchmark_data_smithi088_462306_object51:head# 2024-04-25T09:07:32.567 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c6b2539b:::benchmark_data_smithi088_462306_object254:head# 2024-04-25T09:07:32.568 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c8b8d985:::benchmark_data_smithi088_462306_object118:head# 2024-04-25T09:07:32.569 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c8e1a562:::benchmark_data_smithi088_462306_object245:head# 2024-04-25T09:07:32.570 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cad804f9:::benchmark_data_smithi088_462306_object285:head# 2024-04-25T09:07:32.570 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cb0bb72b:::benchmark_data_smithi088_462306_object268:head# 2024-04-25T09:07:32.571 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cb9c1d39:::benchmark_data_smithi088_462306_object304:head# 2024-04-25T09:07:32.572 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cba404d8:::benchmark_data_smithi088_462306_object34:head# 2024-04-25T09:07:32.572 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cc8e65c8:::benchmark_data_smithi088_462306_object334:head# 2024-04-25T09:07:32.573 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cd75ee52:::benchmark_data_smithi088_462306_object172:head# 2024-04-25T09:07:32.573 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ce72978b:::benchmark_data_smithi088_462306_object226:head# 2024-04-25T09:07:32.574 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cebe6f8b:::benchmark_data_smithi088_462306_object242:head# 2024-04-25T09:07:32.575 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cf135ed8:::benchmark_data_smithi088_462306_object97:head# 2024-04-25T09:07:32.575 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cf1ea7f9:::benchmark_data_smithi088_462306_object337:head# 2024-04-25T09:07:32.576 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cfc2a95a:::benchmark_data_smithi088_462306_object119:head# 2024-04-25T09:07:32.577 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d06b5209:::benchmark_data_smithi088_462306_object152:head# 2024-04-25T09:07:32.577 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d0efe327:::benchmark_data_smithi088_462306_object306:head# 2024-04-25T09:07:32.578 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d182b227:::benchmark_data_smithi088_462306_object100:head# 2024-04-25T09:07:32.578 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d1977b2c:::benchmark_data_smithi088_462306_object166:head# 2024-04-25T09:07:32.578 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d1e00a1c:::benchmark_data_smithi088_462306_object48:head# 2024-04-25T09:07:32.579 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d297305a:::benchmark_data_smithi088_462306_object301:head# 2024-04-25T09:07:32.580 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d2cbd392:::benchmark_data_smithi088_462306_object136:head# 2024-04-25T09:07:32.581 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d2dc1847:::benchmark_data_smithi088_462306_object36:head# 2024-04-25T09:07:32.581 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d2f0530a:::benchmark_data_smithi088_462306_object331:head# 2024-04-25T09:07:32.582 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d41d1c70:::benchmark_data_smithi088_462306_object183:head# 2024-04-25T09:07:32.583 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d42ba1fa:::benchmark_data_smithi088_462306_object38:head# 2024-04-25T09:07:32.583 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d42ca9ea:::benchmark_data_smithi088_462306_object259:head# 2024-04-25T09:07:32.584 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d4ccf510:::benchmark_data_smithi088_462306_object206:head# 2024-04-25T09:07:32.585 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d658cee6:::benchmark_data_smithi088_462306_object279:head# 2024-04-25T09:07:32.585 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d6791640:::benchmark_data_smithi088_462306_object9:head# 2024-04-25T09:07:32.586 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d69229b2:::benchmark_data_smithi088_462306_object312:head# 2024-04-25T09:07:32.587 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d746333f:::benchmark_data_smithi088_462306_object1:head# 2024-04-25T09:07:32.587 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d8820786:::benchmark_data_smithi088_462306_object336:head# 2024-04-25T09:07:32.588 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d8e83737:::benchmark_data_smithi088_462306_object203:head# 2024-04-25T09:07:32.588 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d92e92c0:::benchmark_data_smithi088_462306_object153:head# 2024-04-25T09:07:32.589 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d9ed16a2:::benchmark_data_smithi088_462306_object244:head# 2024-04-25T09:07:32.590 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d9f9d37f:::benchmark_data_smithi088_462306_object322:head# 2024-04-25T09:07:32.590 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:da421750:::benchmark_data_smithi088_462306_object283:head# 2024-04-25T09:07:32.591 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dad3e1cc:::benchmark_data_smithi088_462306_object293:head# 2024-04-25T09:07:32.591 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dbf914c5:::benchmark_data_smithi088_462306_object78:head# 2024-04-25T09:07:32.592 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dbfcbc35:::benchmark_data_smithi088_462306_object41:head# 2024-04-25T09:07:32.593 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dc672535:::benchmark_data_smithi088_462306_object82:head# 2024-04-25T09:07:32.593 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dd09b3db:::benchmark_data_smithi088_462306_object12:head# 2024-04-25T09:07:32.594 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:de8da2f7:::benchmark_data_smithi088_462306_object267:head# 2024-04-25T09:07:32.595 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:df136223:::benchmark_data_smithi088_462306_object81:head# 2024-04-25T09:07:32.595 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:df552722:::benchmark_data_smithi088_462306_object54:head# 2024-04-25T09:07:32.596 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e074f344:::benchmark_data_smithi088_462306_object231:head# 2024-04-25T09:07:32.596 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e0d77fb7:::benchmark_data_smithi088_462306_object229:head# 2024-04-25T09:07:32.597 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e0eb6167:::benchmark_data_smithi088_462306_object101:head# 2024-04-25T09:07:32.598 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e0fbd74b:::benchmark_data_smithi088_462306_object333:head# 2024-04-25T09:07:32.599 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e1956e90:::benchmark_data_smithi088_462306_object243:head# 2024-04-25T09:07:32.599 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e22f836d:::benchmark_data_smithi088_462306_object180:head# 2024-04-25T09:07:32.600 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e25ba6c9:::benchmark_data_smithi088_462306_object15:head# 2024-04-25T09:07:32.601 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e3fbb848:::benchmark_data_smithi088_462306_object30:head# 2024-04-25T09:07:32.601 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e490c252:::benchmark_data_smithi088_462306_object281:head# 2024-04-25T09:07:32.602 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e4fb9482:::benchmark_data_smithi088_462306_object260:head# 2024-04-25T09:07:32.602 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e5dae559:::benchmark_data_smithi088_462306_object170:head# 2024-04-25T09:07:32.603 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e665af06:::benchmark_data_smithi088_462306_object49:head# 2024-04-25T09:07:32.604 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e7b1daf4:::benchmark_data_smithi088_462306_object47:head# 2024-04-25T09:07:32.604 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e9919842:::benchmark_data_smithi088_462306_object289:head# 2024-04-25T09:07:32.605 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ebc23c5b:::benchmark_data_smithi088_462306_object316:head# 2024-04-25T09:07:32.606 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ebddce23:::benchmark_data_smithi088_462306_object286:head# 2024-04-25T09:07:32.606 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f01c7538:::benchmark_data_smithi088_462306_object255:head# 2024-04-25T09:07:32.607 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f03d3cff:::benchmark_data_smithi088_462306_object184:head# 2024-04-25T09:07:32.608 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f0b18d8c:::benchmark_data_smithi088_462306_object7:head# 2024-04-25T09:07:32.609 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f2319ef1:::benchmark_data_smithi088_462306_object46:head# 2024-04-25T09:07:32.609 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f27dc48a:::benchmark_data_smithi088_462306_object175:head# 2024-04-25T09:07:32.610 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f3cf7b95:::benchmark_data_smithi088_462306_object112:head# 2024-04-25T09:07:32.610 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f473732d:::benchmark_data_smithi088_462306_object265:head# 2024-04-25T09:07:32.611 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f50d52f3:::benchmark_data_smithi088_462306_object227:head# 2024-04-25T09:07:32.612 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f69dfeef:::benchmark_data_smithi088_462306_object186:head# 2024-04-25T09:07:32.613 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f7adaaf0:::benchmark_data_smithi088_462306_object214:head# 2024-04-25T09:07:32.613 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f7c73560:::benchmark_data_smithi088_462306_object332:head# 2024-04-25T09:07:32.614 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f8160d49:::benchmark_data_smithi088_462306_object314:head# 2024-04-25T09:07:32.615 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f90ccab7:::benchmark_data_smithi088_462306_object142:head# 2024-04-25T09:07:32.615 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f91a495a:::benchmark_data_smithi088_462306_object72:head# 2024-04-25T09:07:32.616 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fabfa17e:::benchmark_data_smithi088_462306_object140:head# 2024-04-25T09:07:32.617 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fb87786e:::benchmark_data_smithi088_462306_object160:head# 2024-04-25T09:07:32.617 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fbe0687a:::benchmark_data_smithi088_462306_object230:head# 2024-04-25T09:07:32.618 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fc115286:::benchmark_data_smithi088_462306_object39:head# 2024-04-25T09:07:32.619 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fe55e075:::benchmark_data_smithi088_462306_object335:head# 2024-04-25T09:07:32.620 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fefad482:::benchmark_data_smithi088_462306_object79:head# 2024-04-25T09:07:32.620 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ff2bd568:::benchmark_data_smithi088_462306_object90:head# 2024-04-25T09:07:32.621 INFO:tasks.workunit.client.0.smithi088.stderr:Export successful 2024-04-25T09:07:32.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:101: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:07:33.029 INFO:tasks.workunit.client.0.smithi088.stderr:Exporting 1.0 info 1.0( v 8'158 (0'0,8'158] local-lis/les=6/7 n=158 ec=6/6 lis/c=6/6 les/c/f=7/7/0 sis=6) 2024-04-25T09:07:33.031 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0054a4c5:::benchmark_data_smithi088_462306_object56:head# 2024-04-25T09:07:33.031 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0091abb9:::benchmark_data_smithi088_462306_object216:head# 2024-04-25T09:07:33.032 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:01a57ad0:::benchmark_data_smithi088_462306_object317:head# 2024-04-25T09:07:33.032 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:01e683c2:::benchmark_data_smithi088_462306_object106:head# 2024-04-25T09:07:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:021a8ccb:::benchmark_data_smithi088_462306_object108:head# 2024-04-25T09:07:33.033 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:022d1a7b:::benchmark_data_smithi088_462306_object223:head# 2024-04-25T09:07:33.034 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0496a541:::benchmark_data_smithi088_462306_object115:head# 2024-04-25T09:07:33.034 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:052189ea:::benchmark_data_smithi088_462306_object198:head# 2024-04-25T09:07:33.035 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:052d5fd4:::benchmark_data_smithi088_462306_object14:head# 2024-04-25T09:07:33.035 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:06462b6c:::benchmark_data_smithi088_462306_object237:head# 2024-04-25T09:07:33.035 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0665cd86:::benchmark_data_smithi088_462306_object266:head# 2024-04-25T09:07:33.035 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:08ef67f6:::benchmark_data_smithi088_462306_object167:head# 2024-04-25T09:07:33.036 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:09033334:::benchmark_data_smithi088_462306_object57:head# 2024-04-25T09:07:33.036 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0a07ee66:::benchmark_data_smithi088_462306_object139:head# 2024-04-25T09:07:33.037 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0a9dbfd3:::benchmark_data_smithi088_462306_object147:head# 2024-04-25T09:07:33.037 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b31df74:::benchmark_data_smithi088_462306_object103:head# 2024-04-25T09:07:33.037 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b35794e:::benchmark_data_smithi088_462306_object309:head# 2024-04-25T09:07:33.038 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b54d91a:::benchmark_data_smithi088_462306_object143:head# 2024-04-25T09:07:33.038 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b783152:::benchmark_data_smithi088_462306_object53:head# 2024-04-25T09:07:33.039 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0be5222f:::benchmark_data_smithi088_462306_object295:head# 2024-04-25T09:07:33.039 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0c2e7645:::benchmark_data_smithi088_462306_object65:head# 2024-04-25T09:07:33.040 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0d3bf401:::benchmark_data_smithi088_462306_object84:head# 2024-04-25T09:07:33.040 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0da5c762:::benchmark_data_smithi088_462306_object137:head# 2024-04-25T09:07:33.040 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0efc8997:::benchmark_data_smithi088_462306_object228:head# 2024-04-25T09:07:33.041 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0f3cc8d1:::benchmark_data_smithi088_462306_object105:head# 2024-04-25T09:07:33.041 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0f6ddd44:::benchmark_data_smithi088_462306_object131:head# 2024-04-25T09:07:33.042 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:11a8d835:::benchmark_data_smithi088_462306_object20:head# 2024-04-25T09:07:33.042 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:12bf1666:::benchmark_data_smithi088_462306_object63:head# 2024-04-25T09:07:33.042 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1368f5d1:::benchmark_data_smithi088_462306_object208:head# 2024-04-25T09:07:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:13b33233:::benchmark_data_smithi088_462306_object40:head# 2024-04-25T09:07:33.043 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:140e1661:::benchmark_data_smithi088_462306_object247:head# 2024-04-25T09:07:33.044 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:151d580a:::benchmark_data_smithi088_462306_object59:head# 2024-04-25T09:07:33.045 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:16cc41f5:::benchmark_data_smithi088_462306_object67:head# 2024-04-25T09:07:33.045 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:17a2c91b:::benchmark_data_smithi088_462306_object17:head# 2024-04-25T09:07:33.045 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:17f6836e:::benchmark_data_smithi088_462306_object256:head# 2024-04-25T09:07:33.045 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:186d6a7e:::benchmark_data_smithi088_462306_object0:head# 2024-04-25T09:07:33.045 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:19967450:::benchmark_data_smithi088_462306_object123:head# 2024-04-25T09:07:33.045 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1a55af78:::benchmark_data_smithi088_462306_object75:head# 2024-04-25T09:07:33.046 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1acd4c2d:::benchmark_data_smithi088_462306_object187:head# 2024-04-25T09:07:33.046 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1b6403e6:::benchmark_data_smithi088_462306_object33:head# 2024-04-25T09:07:33.046 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1bce90c6:::benchmark_data_smithi088_462306_object217:head# 2024-04-25T09:07:33.047 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1cb6ac0c:::benchmark_data_smithi088_462306_object27:head# 2024-04-25T09:07:33.047 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1cc79080:::benchmark_data_smithi088_462306_object323:head# 2024-04-25T09:07:33.048 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1d5218b3:::benchmark_data_smithi088_462306_object196:head# 2024-04-25T09:07:33.048 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1e94d3aa:::benchmark_data_smithi088_462306_object111:head# 2024-04-25T09:07:33.048 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1eddebf7:::benchmark_data_smithi088_462306_object264:head# 2024-04-25T09:07:33.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1f00585f:::benchmark_data_smithi088_462306_object236:head# 2024-04-25T09:07:33.050 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:20808529:::benchmark_data_smithi088_462306_object151:head# 2024-04-25T09:07:33.050 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:20bdd9c8:::benchmark_data_smithi088_462306_object235:head# 2024-04-25T09:07:33.051 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:20edab27:::benchmark_data_smithi088_462306_object164:head# 2024-04-25T09:07:33.051 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2160d1fc:::benchmark_data_smithi088_462306_object320:head# 2024-04-25T09:07:33.052 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:220a8d5a:::benchmark_data_smithi088_462306_object77:head# 2024-04-25T09:07:33.052 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:231dff78:::benchmark_data_smithi088_462306_object239:head# 2024-04-25T09:07:33.052 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2330fe3c:::benchmark_data_smithi088_462306_object215:head# 2024-04-25T09:07:33.053 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2337b927:::benchmark_data_smithi088_462306_object191:head# 2024-04-25T09:07:33.053 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:23a73227:::benchmark_data_smithi088_462306_object212:head# 2024-04-25T09:07:33.054 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2448079b:::benchmark_data_smithi088_462306_object70:head# 2024-04-25T09:07:33.054 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:27c74952:::benchmark_data_smithi088_462306_object288:head# 2024-04-25T09:07:33.055 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:283568ed:::benchmark_data_smithi088_462306_object315:head# 2024-04-25T09:07:33.055 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:29974a65:::benchmark_data_smithi088_462306_object2:head# 2024-04-25T09:07:33.055 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2bbd0b7d:::benchmark_data_smithi088_462306_object224:head# 2024-04-25T09:07:33.056 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f341493:::benchmark_data_smithi088_462306_object96:head# 2024-04-25T09:07:33.056 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f3e4e35:::benchmark_data_smithi088_462306_object94:head# 2024-04-25T09:07:33.057 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3089f82a:::benchmark_data_smithi088_462306_object270:head# 2024-04-25T09:07:33.057 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:30bc55cf:::benchmark_data_smithi088_462306_object179:head# 2024-04-25T09:07:33.058 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:30ee7b76:::benchmark_data_smithi088_462306_object211:head# 2024-04-25T09:07:33.058 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:324b040a:::benchmark_data_smithi088_462306_object28:head# 2024-04-25T09:07:33.059 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:33bdd052:::benchmark_data_smithi088_462306_object185:head# 2024-04-25T09:07:33.060 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:34a362cf:::benchmark_data_smithi088_462306_object120:head# 2024-04-25T09:07:33.060 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:36ae6861:::benchmark_data_smithi088_462306_object125:head# 2024-04-25T09:07:33.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:37054d0f:::benchmark_data_smithi088_462306_object194:head# 2024-04-25T09:07:33.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:37196b5a:::benchmark_data_smithi088_462306_object32:head# 2024-04-25T09:07:33.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3a0087c7:::benchmark_data_smithi088_462306_object299:head# 2024-04-25T09:07:33.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3bfc2bf9:::benchmark_data_smithi088_462306_object25:head# 2024-04-25T09:07:33.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3c658397:::benchmark_data_smithi088_462306_object114:head# 2024-04-25T09:07:33.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3ca6aa4a:::benchmark_data_smithi088_462306_object292:head# 2024-04-25T09:07:33.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3df4cae2:::benchmark_data_smithi088_462306_object330:head# 2024-04-25T09:07:33.064 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3e25b978:::benchmark_data_smithi088_462306_object250:head# 2024-04-25T09:07:33.065 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3e918cd5:::benchmark_data_smithi088_462306_object287:head# 2024-04-25T09:07:33.065 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3f11fd9d:::benchmark_data_smithi088_462306_object127:head# 2024-04-25T09:07:33.065 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3f85196b:::benchmark_data_smithi088_462306_object135:head# 2024-04-25T09:07:33.065 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:41e0f85b:::benchmark_data_smithi088_462306_object200:head# 2024-04-25T09:07:33.065 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:43088259:::benchmark_data_smithi088_462306_object193:head# 2024-04-25T09:07:33.066 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:452dfc2d:::benchmark_data_smithi088_462306_object144:head# 2024-04-25T09:07:33.066 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:46484682:::benchmark_data_smithi088_462306_object324:head# 2024-04-25T09:07:33.066 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:46c66152:::benchmark_data_smithi088_462306_object149:head# 2024-04-25T09:07:33.066 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:46f14c22:::benchmark_data_smithi088_462306_object300:head# 2024-04-25T09:07:33.067 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4b72756f:::benchmark_data_smithi088_462306_object329:head# 2024-04-25T09:07:33.067 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4ba5396c:::benchmark_data_smithi088_462306_object71:head# 2024-04-25T09:07:33.068 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4c34ed72:::benchmark_data_smithi088_462306_object154:head# 2024-04-25T09:07:33.068 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4c3e8c07:::benchmark_data_smithi088_462306_object221:head# 2024-04-25T09:07:33.068 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4c6252f1:::benchmark_data_smithi088_462306_object18:head# 2024-04-25T09:07:33.069 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4cf34651:::benchmark_data_smithi088_462306_object188:head# 2024-04-25T09:07:33.069 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4d14555f:::benchmark_data_smithi088_462306_object121:head# 2024-04-25T09:07:33.070 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4d5ba4b9:::benchmark_data_smithi088_462306_object158:head# 2024-04-25T09:07:33.070 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4d7ce59d:::benchmark_data_smithi088_462306_object104:head# 2024-04-25T09:07:33.071 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4e3dbe21:::benchmark_data_smithi088_462306_object50:head# 2024-04-25T09:07:33.071 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4e4657de:::benchmark_data_smithi088_462306_object73:head# 2024-04-25T09:07:33.071 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4e9de313:::benchmark_data_smithi088_462306_object93:head# 2024-04-25T09:07:33.072 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:50f6d9da:::benchmark_data_smithi088_462306_object307:head# 2024-04-25T09:07:33.072 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:52f10993:::benchmark_data_smithi088_462306_object205:head# 2024-04-25T09:07:33.073 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:539a6d00:::benchmark_data_smithi088_462306_object107:head# 2024-04-25T09:07:33.074 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:549c3943:::benchmark_data_smithi088_462306_object23:head# 2024-04-25T09:07:33.074 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:55901106:::benchmark_data_smithi088_462306_object89:head# 2024-04-25T09:07:33.074 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:56f25dc9:::benchmark_data_smithi088_462306_object68:head# 2024-04-25T09:07:33.075 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:56f2d5c4:::benchmark_data_smithi088_462306_object274:head# 2024-04-25T09:07:33.075 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:58ffc1f4:::benchmark_data_smithi088_462306_object124:head# 2024-04-25T09:07:33.076 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:59107f03:::benchmark_data_smithi088_462306_object157:head# 2024-04-25T09:07:33.076 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5933a9b4:::benchmark_data_smithi088_462306_object4:head# 2024-04-25T09:07:33.077 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:594172d7:::benchmark_data_smithi088_462306_object95:head# 2024-04-25T09:07:33.077 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5abcca99:::benchmark_data_smithi088_462306_object85:head# 2024-04-25T09:07:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5b39136e:::benchmark_data_smithi088_462306_object168:head# 2024-04-25T09:07:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5bbb1972:::benchmark_data_smithi088_462306_object113:head# 2024-04-25T09:07:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5c0a9a1b:::benchmark_data_smithi088_462306_object80:head# 2024-04-25T09:07:33.078 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5c15d000:::benchmark_data_smithi088_462306_object318:head# 2024-04-25T09:07:33.079 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5c895488:::benchmark_data_smithi088_462306_object58:head# 2024-04-25T09:07:33.079 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5cffb09b:::benchmark_data_smithi088_462306_object31:head# 2024-04-25T09:07:33.080 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5daa5792:::benchmark_data_smithi088_462306_object192:head# 2024-04-25T09:07:33.080 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5e669d4b:::benchmark_data_smithi088_462306_object248:head# 2024-04-25T09:07:33.081 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5e753d70:::benchmark_data_smithi088_462306_object88:head# 2024-04-25T09:07:33.081 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5fc0d91a:::benchmark_data_smithi088_462306_object83:head# 2024-04-25T09:07:33.082 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:60c47461:::benchmark_data_smithi088_462306_object201:head# 2024-04-25T09:07:33.082 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:62524425:::benchmark_data_smithi088_462306_object284:head# 2024-04-25T09:07:33.083 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:62e454a3:::benchmark_data_smithi088_462306_object10:head# 2024-04-25T09:07:33.083 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:62e744cb:::benchmark_data_smithi088_462306_object297:head# 2024-04-25T09:07:33.083 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:63a33891:::benchmark_data_smithi088_462306_object102:head# 2024-04-25T09:07:33.084 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:64acde59:::benchmark_data_smithi088_462306_object321:head# 2024-04-25T09:07:33.084 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:64ca7391:::benchmark_data_smithi088_462306_object213:head# 2024-04-25T09:07:33.085 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:65bc1817:::benchmark_data_smithi088_462306_object303:head# 2024-04-25T09:07:33.085 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6633f92d:::benchmark_data_smithi088_462306_object126:head# 2024-04-25T09:07:33.086 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:678dbd08:::benchmark_data_smithi088_462306_object52:head# 2024-04-25T09:07:33.086 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:67a6186b:::benchmark_data_smithi088_462306_object327:head# 2024-04-25T09:07:33.086 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:67aa77b1:::benchmark_data_smithi088_462306_object44:head# 2024-04-25T09:07:33.087 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6806c11f:::benchmark_data_smithi088_462306_object261:head# 2024-04-25T09:07:33.087 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:68b0f3d0:::benchmark_data_smithi088_462306_object328:head# 2024-04-25T09:07:33.088 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6a010110:::benchmark_data_smithi088_462306_object296:head# 2024-04-25T09:07:33.088 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6b5538cb:::benchmark_data_smithi088_462306_object308:head# 2024-04-25T09:07:33.088 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6b7b59b0:::benchmark_data_smithi088_462306_object218:head# 2024-04-25T09:07:33.089 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6bbd025e:::benchmark_data_smithi088_462306_object141:head# 2024-04-25T09:07:33.089 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6c16e74e:::benchmark_data_smithi088_462306_object241:head# 2024-04-25T09:07:33.090 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6d493727:::benchmark_data_smithi088_462306_object86:head# 2024-04-25T09:07:33.090 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7041d2eb:::benchmark_data_smithi088_462306_object171:head# 2024-04-25T09:07:33.091 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:72ab0865:::benchmark_data_smithi088_462306_object42:head# 2024-04-25T09:07:33.091 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:73e18593:::benchmark_data_smithi088_462306_object234:head# 2024-04-25T09:07:33.092 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7411cf6e:::benchmark_data_smithi088_462306_object305:head# 2024-04-25T09:07:33.092 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:749c107b:::benchmark_data_smithi088_462306_object29:head# 2024-04-25T09:07:33.093 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7759d088:::benchmark_data_smithi088_462306_object310:head# 2024-04-25T09:07:33.093 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:777330ea:::benchmark_data_smithi088_462306_object262:head# 2024-04-25T09:07:33.094 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:77982bca:::benchmark_data_smithi088_462306_object269:head# 2024-04-25T09:07:33.094 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7879b0d4:::benchmark_data_smithi088_462306_object313:head# 2024-04-25T09:07:33.095 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7c39dbd4:::benchmark_data_smithi088_462306_object45:head# 2024-04-25T09:07:33.095 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7cb590f7:::benchmark_data_smithi088_462306_object177:head# 2024-04-25T09:07:33.095 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7d29cf47:::benchmark_data_smithi088_462306_object249:head# 2024-04-25T09:07:33.096 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7db41ef9:::benchmark_data_smithi088_462306_object190:head# 2024-04-25T09:07:33.097 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7de60352:::benchmark_data_smithi088_462306_object210:head# 2024-04-25T09:07:33.097 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7e59f398:::benchmark_data_smithi088_462306_object110:head# 2024-04-25T09:07:33.097 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7efa7304:::benchmark_data_smithi088_462306_object74:head# 2024-04-25T09:07:33.098 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7fd54788:::benchmark_data_smithi088_462306_object133:head# 2024-04-25T09:07:33.098 INFO:tasks.workunit.client.0.smithi088.stderr:Export successful 2024-04-25T09:07:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:102: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:07:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:07:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:07:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:07:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:07:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:07:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:07:33.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:33.398 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:07:33.400 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:07:33.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:07:33.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:07:33.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:07:33.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:07:33.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:07:33.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:07:33.400 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:07:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:07:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:07:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:07:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:07:33.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:07:33.401 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:07:33.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:07:33.402 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:07:33.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:07:33.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:07:33.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:07:33.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:07:33.436 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:07:33.447+0000 7f5c2deb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:33.439 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:07:33.451+0000 7f5c2deb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:33.440 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:07:33.451+0000 7f5c2deb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:33.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:07:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:07:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:07:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:07:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:07:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:33.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:07:33.885 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:07:33.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:33.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:34.004 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:07:34.015+0000 7f5c2deb27c0 -1 Falling back to public interface 2024-04-25T09:07:34.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:35.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:35.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:35.317 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:07:35.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:07:35.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:35.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:35.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:36.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:36.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:36.736 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:07:36.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:07:36.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:36.736 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:37.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:37.236 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:07:37.247+0000 7f5c2deb27c0 -1 osd.0 8 log_to_monitors true 2024-04-25T09:07:38.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:38.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:38.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:07:38.158 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:07:38.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:38.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:38.662 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 11 up_thru 11 down_at 9 last_clean_interval [5,8) [v2:127.0.0.1:6802/4190162881,v1:127.0.0.1:6803/4190162881] [v2:127.0.0.1:6804/4190162881,v1:127.0.0.1:6805/4190162881] exists,up 168d882f-4939-4337-8b16-26a6cead1b6d 2024-04-25T09:07:38.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:07:38.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:07:38.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:07:38.663 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:104: TEST_import_after_merge_and_gap: ceph osd pool set foo pg_num 1 2024-04-25T09:07:39.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:105: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:07:44.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: ceph daemon osd.0 perf dump 2024-04-25T09:07:44.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: jq .osd.numpg 2024-04-25T09:07:44.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: grep 2 2024-04-25T09:07:44.462 INFO:tasks.workunit.client.0.smithi088.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-25T09:07:44.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:107: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:07:44.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:07:44.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:07:44.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:07:44.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:07:44.466 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:07:44.466 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:07:44.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:07:44.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:07:44.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:07:44.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:07:44.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:07:44.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:07:44.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:07:44.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:07:44.569 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:07:44.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:07:44.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:07:44.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:44.994 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:07:45.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=47244640260 2024-04-25T09:07:45.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 47244640260 2024-04-25T09:07:45.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-47244640260' 2024-04-25T09:07:45.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:45.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-47244640260 2024-04-25T09:07:45.237 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:45.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:07:45.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-47244640260 2024-04-25T09:07:45.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:45.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=47244640260 2024-04-25T09:07:45.240 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 47244640260 2024-04-25T09:07:45.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 47244640260' 2024-04-25T09:07:45.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:07:45.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 47244640258 -lt 47244640260 2024-04-25T09:07:45.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:07:46.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:07:46.650 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:07:47.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 47244640260 -lt 47244640260 2024-04-25T09:07:47.281 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:07:47.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:07:47.282 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:07:47.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:07:47.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:07:47.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:07:47.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:07:47.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:07:47.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:07:47.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:07:47.854 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:07:48.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:07:48.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:07:48.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:07:48.254 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:07:48.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:07:48.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:07:48.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:07:48.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:110: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:07:48.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:07:48.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:07:48.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:07:48.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:07:48.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:07:49.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:07:49.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:111: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:07:50.626 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T09:07:50.627 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T09:07:50.627 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:07:51.094 INFO:tasks.workunit.client.0.smithi088.stdout:Remove successful 2024-04-25T09:07:51.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:113: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:07:52.397 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e19 pg_nums {1={6=2,18=1}} deleted_pools ) 2024-04-25T09:07:52.398 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.1 2024-04-25T09:07:52.398 INFO:tasks.workunit.client.0.smithi088.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:07:52.399 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81099aed:::benchmark_data_smithi088_462306_object181:head# 2024-04-25T09:07:52.399 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.581 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81ae27f9:::benchmark_data_smithi088_462306_object252:head# 2024-04-25T09:07:52.581 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.622 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81b12e8e:::benchmark_data_smithi088_462306_object161:head# 2024-04-25T09:07:52.623 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.664 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81e3c822:::benchmark_data_smithi088_462306_object64:head# 2024-04-25T09:07:52.664 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.706 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:82743540:::benchmark_data_smithi088_462306_object276:head# 2024-04-25T09:07:52.706 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.748 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:842cb2a9:::benchmark_data_smithi088_462306_object278:head# 2024-04-25T09:07:52.748 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.789 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:84c54114:::benchmark_data_smithi088_462306_object66:head# 2024-04-25T09:07:52.790 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.831 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:85078920:::benchmark_data_smithi088_462306_object62:head# 2024-04-25T09:07:52.831 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.873 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:85f7c2db:::benchmark_data_smithi088_462306_object326:head# 2024-04-25T09:07:52.873 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.915 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8770f696:::benchmark_data_smithi088_462306_object253:head# 2024-04-25T09:07:52.915 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.957 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87b0211e:::benchmark_data_smithi088_462306_object8:head# 2024-04-25T09:07:52.957 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:52.998 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87cd1abd:::benchmark_data_smithi088_462306_object128:head# 2024-04-25T09:07:52.999 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.048 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87cf274d:::benchmark_data_smithi088_462306_object19:head# 2024-04-25T09:07:53.049 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.090 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87f67271:::benchmark_data_smithi088_462306_object189:head# 2024-04-25T09:07:53.091 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.123 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:88f3d2ff:::benchmark_data_smithi088_462306_object173:head# 2024-04-25T09:07:53.124 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.157 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8a4612ce:::benchmark_data_smithi088_462306_object233:head# 2024-04-25T09:07:53.157 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.190 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8bb6f309:::benchmark_data_smithi088_462306_object132:head# 2024-04-25T09:07:53.190 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.224 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8ca22c61:::benchmark_data_smithi088_462306_object220:head# 2024-04-25T09:07:53.224 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.266 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8d801fc0:::benchmark_data_smithi088_462306_object207:head# 2024-04-25T09:07:53.266 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.299 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8fbb559d:::benchmark_data_smithi088_462306_object257:head# 2024-04-25T09:07:53.299 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.333 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91623a25:::benchmark_data_smithi088_462306_object138:head# 2024-04-25T09:07:53.333 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.366 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9196f898:::benchmark_data_smithi088_462306_object6:head# 2024-04-25T09:07:53.366 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.400 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91e42837:::benchmark_data_smithi088_462306_object176:head# 2024-04-25T09:07:53.400 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.433 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91e98fd5:::benchmark_data_smithi088_462306_object35:head# 2024-04-25T09:07:53.433 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.467 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92220cfe:::benchmark_data_smithi088_462306_object22:head# 2024-04-25T09:07:53.467 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.500 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9265404e:::benchmark_data_smithi088_462306_object169:head# 2024-04-25T09:07:53.500 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.533 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92ad4e27:::benchmark_data_smithi088_462306_object238:head# 2024-04-25T09:07:53.534 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.567 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:934c91ea:::benchmark_data_smithi088_462306_object246:head# 2024-04-25T09:07:53.567 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.600 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:937e9a49:::benchmark_data_smithi088_462306_object199:head# 2024-04-25T09:07:53.600 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.634 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:93b8c46b:::benchmark_data_smithi088_462306_object76:head# 2024-04-25T09:07:53.634 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.667 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9411c558:::benchmark_data_smithi088_462306_object24:head# 2024-04-25T09:07:53.667 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.701 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:94abe535:::benchmark_data_smithi088_462306_object60:head# 2024-04-25T09:07:53.701 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.734 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:96cf9e3a:::benchmark_data_smithi088_462306_object325:head# 2024-04-25T09:07:53.734 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.768 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:98888887:::benchmark_data_smithi088_462306_object130:head# 2024-04-25T09:07:53.768 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.801 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:98cc271a:::benchmark_data_smithi088_462306_object174:head# 2024-04-25T09:07:53.801 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.835 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9935b1ed:::benchmark_data_smithi088_462306_object272:head# 2024-04-25T09:07:53.835 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.868 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9938cea0:::benchmark_data_smithi088_462306_object162:head# 2024-04-25T09:07:53.868 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.935 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9acb5316:::benchmark_data_smithi088_462306_object222:head# 2024-04-25T09:07:53.935 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:53.985 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9b089756:::benchmark_data_smithi088_462306_object271:head# 2024-04-25T09:07:53.985 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.019 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9baeb73f:::benchmark_data_smithi088_462306_object117:head# 2024-04-25T09:07:54.019 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.052 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9bf57a33:::benchmark_data_smithi088_462306_object204:head# 2024-04-25T09:07:54.052 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.086 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9c595d90:::benchmark_data_smithi088_462306_object16:head# 2024-04-25T09:07:54.086 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.127 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9caedf93:::benchmark_data_smithi088_462306_object109:head# 2024-04-25T09:07:54.128 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.169 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9cb158b2:::benchmark_data_smithi088_462306_object150:head# 2024-04-25T09:07:54.169 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.211 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9f6e371a:::benchmark_data_smithi088_462306_object275:head# 2024-04-25T09:07:54.211 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.253 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a20735fd:::benchmark_data_smithi088_462306_object282:head# 2024-04-25T09:07:54.253 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.295 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a2c8e7c0:::benchmark_data_smithi088_462306_object116:head# 2024-04-25T09:07:54.295 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.336 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a3548fec:::benchmark_data_smithi088_462306_object178:head# 2024-04-25T09:07:54.336 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.384 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a3d97e4f:::benchmark_data_smithi088_462306_object37:head# 2024-04-25T09:07:54.384 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.466 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a40b440b:::benchmark_data_smithi088_462306_object197:head# 2024-04-25T09:07:54.466 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.529 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a4fe7d56:::benchmark_data_smithi088_462306_object92:head# 2024-04-25T09:07:54.529 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.571 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6c17891:::benchmark_data_smithi088_462306_object146:head# 2024-04-25T09:07:54.571 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.613 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6cdc671:::benchmark_data_smithi088_462306_object195:head# 2024-04-25T09:07:54.613 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.655 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6ebe5bf:::benchmark_data_smithi088_462306_object145:head# 2024-04-25T09:07:54.655 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.696 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a7506a6e:::benchmark_data_smithi088_462306_object122:head# 2024-04-25T09:07:54.696 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.738 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a87a78a1:::benchmark_data_smithi088_462306_object5:head# 2024-04-25T09:07:54.738 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.780 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a87bb1e0:::benchmark_data_smithi088_462306_object263:head# 2024-04-25T09:07:54.780 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.821 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a8de3fc3:::benchmark_data_smithi088_462306_object134:head# 2024-04-25T09:07:54.822 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.863 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a92036b2:::benchmark_data_smithi088_462306_object225:head# 2024-04-25T09:07:54.864 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.905 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:aa84cf26:::benchmark_data_smithi088_462306_object11:head# 2024-04-25T09:07:54.905 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.947 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab034bbd:::benchmark_data_smithi088_462306_object232:head# 2024-04-25T09:07:54.947 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:54.989 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab565d7b:::benchmark_data_smithi088_462306_object302:head# 2024-04-25T09:07:54.989 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.030 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:abc81ff6:::benchmark_data_smithi088_462306_object69:head# 2024-04-25T09:07:55.031 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.072 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ac08c85b:::benchmark_data_smithi088_462306_object61:head# 2024-04-25T09:07:55.109 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.114 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:acecab62:::benchmark_data_smithi088_462306_object219:head# 2024-04-25T09:07:55.114 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.164 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ad48ca91:::benchmark_data_smithi088_462306_object209:head# 2024-04-25T09:07:55.164 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.206 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:af94d379:::benchmark_data_smithi088_462306_object298:head# 2024-04-25T09:07:55.206 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.248 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b06befdd:::benchmark_data_smithi088_462306_object156:head# 2024-04-25T09:07:55.248 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.290 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b0c2cf35:::benchmark_data_smithi088_462306_object240:head# 2024-04-25T09:07:55.290 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.331 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b17d8fef:::benchmark_data_smithi088_462306_object13:head# 2024-04-25T09:07:55.332 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.382 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b20bd2ac:::benchmark_data_smithi088_462306_object26:head# 2024-04-25T09:07:55.382 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.423 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b2160c47:::benchmark_data_smithi088_462306_object159:head# 2024-04-25T09:07:55.423 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.465 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b2f527bb:::benchmark_data_smithi088_462306_object21:head# 2024-04-25T09:07:55.465 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.507 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b38816db:::benchmark_data_smithi088_462306_object87:head# 2024-04-25T09:07:55.507 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.557 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b39555ce:::benchmark_data_smithi088_462306_object311:head# 2024-04-25T09:07:55.557 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.599 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b3d4574c:::benchmark_data_smithi088_462306_object182:head# 2024-04-25T09:07:55.599 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.632 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b50b3477:::benchmark_data_smithi088_462306_object55:head# 2024-04-25T09:07:55.633 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.666 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b7cd0d37:::benchmark_data_smithi088_462306_object294:head# 2024-04-25T09:07:55.666 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.699 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b82e2947:::benchmark_data_smithi088_462306_object163:head# 2024-04-25T09:07:55.699 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.733 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:07:55.733 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.766 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b8db97a9:::benchmark_data_smithi088_462306_object165:head# 2024-04-25T09:07:55.766 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.800 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b97bb4a3:::benchmark_data_smithi088_462306_object319:head# 2024-04-25T09:07:55.800 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.833 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba2c6dac:::benchmark_data_smithi088_462306_object251:head# 2024-04-25T09:07:55.834 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.867 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba685cfd:::benchmark_data_smithi088_462306_object99:head# 2024-04-25T09:07:55.867 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.942 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba7c5b40:::benchmark_data_smithi088_462306_object202:head# 2024-04-25T09:07:55.942 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:55.992 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba8d78f2:::benchmark_data_smithi088_462306_object148:head# 2024-04-25T09:07:55.992 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.026 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:baf95fb7:::benchmark_data_smithi088_462306_object290:head# 2024-04-25T09:07:56.026 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.059 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bca58a03:::benchmark_data_smithi088_462306_object129:head# 2024-04-25T09:07:56.059 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.093 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bdd50353:::benchmark_data_smithi088_462306_object98:head# 2024-04-25T09:07:56.093 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.151 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:be02a273:::benchmark_data_smithi088_462306_object291:head# 2024-04-25T09:07:56.151 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.184 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bfa07270:::benchmark_data_smithi088_462306_object258:head# 2024-04-25T09:07:56.185 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.218 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bfc2ddbb:::benchmark_data_smithi088_462306_object155:head# 2024-04-25T09:07:56.218 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.252 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c18f632f:::benchmark_data_smithi088_462306_object43:head# 2024-04-25T09:07:56.252 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.285 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c2f3461e:::benchmark_data_smithi088_462306_object91:head# 2024-04-25T09:07:56.285 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.318 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c40c5699:::benchmark_data_smithi088_462306_object280:head# 2024-04-25T09:07:56.319 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.352 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c4517802:::benchmark_data_smithi088_462306_object3:head# 2024-04-25T09:07:56.352 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.419 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c4dec443:::benchmark_data_smithi088_462306_object277:head# 2024-04-25T09:07:56.419 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.478 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c588688f:::benchmark_data_smithi088_462306_object273:head# 2024-04-25T09:07:56.478 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.544 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c621a2c5:::benchmark_data_smithi088_462306_object51:head# 2024-04-25T09:07:56.545 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.578 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c6b2539b:::benchmark_data_smithi088_462306_object254:head# 2024-04-25T09:07:56.578 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.611 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c8b8d985:::benchmark_data_smithi088_462306_object118:head# 2024-04-25T09:07:56.611 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.645 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c8e1a562:::benchmark_data_smithi088_462306_object245:head# 2024-04-25T09:07:56.645 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.687 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cad804f9:::benchmark_data_smithi088_462306_object285:head# 2024-04-25T09:07:56.687 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.728 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cb0bb72b:::benchmark_data_smithi088_462306_object268:head# 2024-04-25T09:07:56.729 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.795 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cb9c1d39:::benchmark_data_smithi088_462306_object304:head# 2024-04-25T09:07:56.796 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.837 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cba404d8:::benchmark_data_smithi088_462306_object34:head# 2024-04-25T09:07:56.837 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.879 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cc8e65c8:::benchmark_data_smithi088_462306_object334:head# 2024-04-25T09:07:56.879 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.921 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cd75ee52:::benchmark_data_smithi088_462306_object172:head# 2024-04-25T09:07:56.921 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:56.962 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ce72978b:::benchmark_data_smithi088_462306_object226:head# 2024-04-25T09:07:56.963 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.004 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cebe6f8b:::benchmark_data_smithi088_462306_object242:head# 2024-04-25T09:07:57.004 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.046 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cf135ed8:::benchmark_data_smithi088_462306_object97:head# 2024-04-25T09:07:57.046 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.088 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cf1ea7f9:::benchmark_data_smithi088_462306_object337:head# 2024-04-25T09:07:57.088 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.129 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cfc2a95a:::benchmark_data_smithi088_462306_object119:head# 2024-04-25T09:07:57.130 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.171 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d06b5209:::benchmark_data_smithi088_462306_object152:head# 2024-04-25T09:07:57.172 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.213 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d0efe327:::benchmark_data_smithi088_462306_object306:head# 2024-04-25T09:07:57.213 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.255 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d182b227:::benchmark_data_smithi088_462306_object100:head# 2024-04-25T09:07:57.255 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.297 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d1977b2c:::benchmark_data_smithi088_462306_object166:head# 2024-04-25T09:07:57.297 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.339 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d1e00a1c:::benchmark_data_smithi088_462306_object48:head# 2024-04-25T09:07:57.339 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.380 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d297305a:::benchmark_data_smithi088_462306_object301:head# 2024-04-25T09:07:57.381 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.422 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2cbd392:::benchmark_data_smithi088_462306_object136:head# 2024-04-25T09:07:57.422 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.464 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2dc1847:::benchmark_data_smithi088_462306_object36:head# 2024-04-25T09:07:57.464 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.506 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2f0530a:::benchmark_data_smithi088_462306_object331:head# 2024-04-25T09:07:57.506 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.548 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d41d1c70:::benchmark_data_smithi088_462306_object183:head# 2024-04-25T09:07:57.548 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.589 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d42ba1fa:::benchmark_data_smithi088_462306_object38:head# 2024-04-25T09:07:57.590 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.631 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d42ca9ea:::benchmark_data_smithi088_462306_object259:head# 2024-04-25T09:07:57.631 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.673 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d4ccf510:::benchmark_data_smithi088_462306_object206:head# 2024-04-25T09:07:57.673 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.715 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d658cee6:::benchmark_data_smithi088_462306_object279:head# 2024-04-25T09:07:57.715 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.756 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d6791640:::benchmark_data_smithi088_462306_object9:head# 2024-04-25T09:07:57.757 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.798 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d69229b2:::benchmark_data_smithi088_462306_object312:head# 2024-04-25T09:07:57.798 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.848 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d746333f:::benchmark_data_smithi088_462306_object1:head# 2024-04-25T09:07:57.848 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.909 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d8820786:::benchmark_data_smithi088_462306_object336:head# 2024-04-25T09:07:57.909 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:57.982 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d8e83737:::benchmark_data_smithi088_462306_object203:head# 2024-04-25T09:07:57.983 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.024 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d92e92c0:::benchmark_data_smithi088_462306_object153:head# 2024-04-25T09:07:58.024 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.066 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d9ed16a2:::benchmark_data_smithi088_462306_object244:head# 2024-04-25T09:07:58.066 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.108 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d9f9d37f:::benchmark_data_smithi088_462306_object322:head# 2024-04-25T09:07:58.108 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.150 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:da421750:::benchmark_data_smithi088_462306_object283:head# 2024-04-25T09:07:58.150 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.191 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dad3e1cc:::benchmark_data_smithi088_462306_object293:head# 2024-04-25T09:07:58.191 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.233 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dbf914c5:::benchmark_data_smithi088_462306_object78:head# 2024-04-25T09:07:58.233 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.275 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dbfcbc35:::benchmark_data_smithi088_462306_object41:head# 2024-04-25T09:07:58.275 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.309 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dc672535:::benchmark_data_smithi088_462306_object82:head# 2024-04-25T09:07:58.309 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.342 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd09b3db:::benchmark_data_smithi088_462306_object12:head# 2024-04-25T09:07:58.342 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.383 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:de8da2f7:::benchmark_data_smithi088_462306_object267:head# 2024-04-25T09:07:58.384 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.434 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:df136223:::benchmark_data_smithi088_462306_object81:head# 2024-04-25T09:07:58.434 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.501 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:df552722:::benchmark_data_smithi088_462306_object54:head# 2024-04-25T09:07:58.501 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.568 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e074f344:::benchmark_data_smithi088_462306_object231:head# 2024-04-25T09:07:58.568 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.601 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0d77fb7:::benchmark_data_smithi088_462306_object229:head# 2024-04-25T09:07:58.602 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.635 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0eb6167:::benchmark_data_smithi088_462306_object101:head# 2024-04-25T09:07:58.635 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.668 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0fbd74b:::benchmark_data_smithi088_462306_object333:head# 2024-04-25T09:07:58.668 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.702 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e1956e90:::benchmark_data_smithi088_462306_object243:head# 2024-04-25T09:07:58.702 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.735 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e22f836d:::benchmark_data_smithi088_462306_object180:head# 2024-04-25T09:07:58.735 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.769 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e25ba6c9:::benchmark_data_smithi088_462306_object15:head# 2024-04-25T09:07:58.769 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.802 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e3fbb848:::benchmark_data_smithi088_462306_object30:head# 2024-04-25T09:07:58.802 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.836 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e490c252:::benchmark_data_smithi088_462306_object281:head# 2024-04-25T09:07:58.836 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.869 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e4fb9482:::benchmark_data_smithi088_462306_object260:head# 2024-04-25T09:07:58.869 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.903 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e5dae559:::benchmark_data_smithi088_462306_object170:head# 2024-04-25T09:07:58.903 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.936 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e665af06:::benchmark_data_smithi088_462306_object49:head# 2024-04-25T09:07:58.936 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:58.970 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e7b1daf4:::benchmark_data_smithi088_462306_object47:head# 2024-04-25T09:07:58.970 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.003 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e9919842:::benchmark_data_smithi088_462306_object289:head# 2024-04-25T09:07:59.003 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.036 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ebc23c5b:::benchmark_data_smithi088_462306_object316:head# 2024-04-25T09:07:59.036 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.070 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ebddce23:::benchmark_data_smithi088_462306_object286:head# 2024-04-25T09:07:59.070 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.104 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f01c7538:::benchmark_data_smithi088_462306_object255:head# 2024-04-25T09:07:59.104 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.137 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f03d3cff:::benchmark_data_smithi088_462306_object184:head# 2024-04-25T09:07:59.137 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.170 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f0b18d8c:::benchmark_data_smithi088_462306_object7:head# 2024-04-25T09:07:59.170 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.204 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f2319ef1:::benchmark_data_smithi088_462306_object46:head# 2024-04-25T09:07:59.204 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.237 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f27dc48a:::benchmark_data_smithi088_462306_object175:head# 2024-04-25T09:07:59.237 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.271 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f3cf7b95:::benchmark_data_smithi088_462306_object112:head# 2024-04-25T09:07:59.271 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.304 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f473732d:::benchmark_data_smithi088_462306_object265:head# 2024-04-25T09:07:59.304 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.338 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f50d52f3:::benchmark_data_smithi088_462306_object227:head# 2024-04-25T09:07:59.338 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.371 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f69dfeef:::benchmark_data_smithi088_462306_object186:head# 2024-04-25T09:07:59.371 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.413 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f7adaaf0:::benchmark_data_smithi088_462306_object214:head# 2024-04-25T09:07:59.413 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.455 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f7c73560:::benchmark_data_smithi088_462306_object332:head# 2024-04-25T09:07:59.455 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.496 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f8160d49:::benchmark_data_smithi088_462306_object314:head# 2024-04-25T09:07:59.496 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.538 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f90ccab7:::benchmark_data_smithi088_462306_object142:head# 2024-04-25T09:07:59.538 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.580 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f91a495a:::benchmark_data_smithi088_462306_object72:head# 2024-04-25T09:07:59.580 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.622 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fabfa17e:::benchmark_data_smithi088_462306_object140:head# 2024-04-25T09:07:59.622 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.672 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fb87786e:::benchmark_data_smithi088_462306_object160:head# 2024-04-25T09:07:59.672 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.739 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fbe0687a:::benchmark_data_smithi088_462306_object230:head# 2024-04-25T09:07:59.739 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.781 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fc115286:::benchmark_data_smithi088_462306_object39:head# 2024-04-25T09:07:59.781 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.822 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fe55e075:::benchmark_data_smithi088_462306_object335:head# 2024-04-25T09:07:59.823 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.864 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fefad482:::benchmark_data_smithi088_462306_object79:head# 2024-04-25T09:07:59.864 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.939 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ff2bd568:::benchmark_data_smithi088_462306_object90:head# 2024-04-25T09:07:59.939 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:07:59.990 INFO:tasks.workunit.client.0.smithi088.stdout:write_pg epoch 8 info 1.1( v 8'181 (0'0,8'181] local-lis/les=6/7 n=181 ec=6/6 lis/c=6/6 les/c/f=7/7/0 sis=6) 2024-04-25T09:08:00.032 INFO:tasks.workunit.client.0.smithi088.stdout:Import successful 2024-04-25T09:08:00.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:114: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:08:01.987 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e19 pg_nums {1={6=2,18=1}} deleted_pools ) 2024-04-25T09:08:01.987 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.0 2024-04-25T09:08:01.987 INFO:tasks.workunit.client.0.smithi088.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:08:01.987 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0054a4c5:::benchmark_data_smithi088_462306_object56:head# 2024-04-25T09:08:01.988 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.062 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0091abb9:::benchmark_data_smithi088_462306_object216:head# 2024-04-25T09:08:02.062 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.095 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:01a57ad0:::benchmark_data_smithi088_462306_object317:head# 2024-04-25T09:08:02.095 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.129 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:01e683c2:::benchmark_data_smithi088_462306_object106:head# 2024-04-25T09:08:02.129 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.162 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:021a8ccb:::benchmark_data_smithi088_462306_object108:head# 2024-04-25T09:08:02.162 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.196 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:022d1a7b:::benchmark_data_smithi088_462306_object223:head# 2024-04-25T09:08:02.196 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.229 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0496a541:::benchmark_data_smithi088_462306_object115:head# 2024-04-25T09:08:02.229 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.262 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:052189ea:::benchmark_data_smithi088_462306_object198:head# 2024-04-25T09:08:02.263 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.296 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:052d5fd4:::benchmark_data_smithi088_462306_object14:head# 2024-04-25T09:08:02.296 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.329 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:06462b6c:::benchmark_data_smithi088_462306_object237:head# 2024-04-25T09:08:02.330 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.363 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0665cd86:::benchmark_data_smithi088_462306_object266:head# 2024-04-25T09:08:02.363 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.430 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:08ef67f6:::benchmark_data_smithi088_462306_object167:head# 2024-04-25T09:08:02.430 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.497 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:09033334:::benchmark_data_smithi088_462306_object57:head# 2024-04-25T09:08:02.497 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.547 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a07ee66:::benchmark_data_smithi088_462306_object139:head# 2024-04-25T09:08:02.547 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.581 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a9dbfd3:::benchmark_data_smithi088_462306_object147:head# 2024-04-25T09:08:02.581 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.631 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b31df74:::benchmark_data_smithi088_462306_object103:head# 2024-04-25T09:08:02.631 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.664 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b35794e:::benchmark_data_smithi088_462306_object309:head# 2024-04-25T09:08:02.664 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.698 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b54d91a:::benchmark_data_smithi088_462306_object143:head# 2024-04-25T09:08:02.698 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.731 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b783152:::benchmark_data_smithi088_462306_object53:head# 2024-04-25T09:08:02.731 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.764 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0be5222f:::benchmark_data_smithi088_462306_object295:head# 2024-04-25T09:08:02.765 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.807 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0c2e7645:::benchmark_data_smithi088_462306_object65:head# 2024-04-25T09:08:02.807 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.840 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0d3bf401:::benchmark_data_smithi088_462306_object84:head# 2024-04-25T09:08:02.840 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.874 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0da5c762:::benchmark_data_smithi088_462306_object137:head# 2024-04-25T09:08:02.874 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.907 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0efc8997:::benchmark_data_smithi088_462306_object228:head# 2024-04-25T09:08:02.907 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.940 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0f3cc8d1:::benchmark_data_smithi088_462306_object105:head# 2024-04-25T09:08:02.940 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:02.974 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0f6ddd44:::benchmark_data_smithi088_462306_object131:head# 2024-04-25T09:08:02.974 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.007 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:11a8d835:::benchmark_data_smithi088_462306_object20:head# 2024-04-25T09:08:03.007 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.041 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:12bf1666:::benchmark_data_smithi088_462306_object63:head# 2024-04-25T09:08:03.041 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.074 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1368f5d1:::benchmark_data_smithi088_462306_object208:head# 2024-04-25T09:08:03.074 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.108 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:13b33233:::benchmark_data_smithi088_462306_object40:head# 2024-04-25T09:08:03.108 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.141 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:140e1661:::benchmark_data_smithi088_462306_object247:head# 2024-04-25T09:08:03.141 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.175 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:151d580a:::benchmark_data_smithi088_462306_object59:head# 2024-04-25T09:08:03.175 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.208 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:16cc41f5:::benchmark_data_smithi088_462306_object67:head# 2024-04-25T09:08:03.208 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.242 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17a2c91b:::benchmark_data_smithi088_462306_object17:head# 2024-04-25T09:08:03.242 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.298 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17f6836e:::benchmark_data_smithi088_462306_object256:head# 2024-04-25T09:08:03.298 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.333 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:186d6a7e:::benchmark_data_smithi088_462306_object0:head# 2024-04-25T09:08:03.334 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.367 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:19967450:::benchmark_data_smithi088_462306_object123:head# 2024-04-25T09:08:03.367 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.409 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1a55af78:::benchmark_data_smithi088_462306_object75:head# 2024-04-25T09:08:03.409 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.450 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1acd4c2d:::benchmark_data_smithi088_462306_object187:head# 2024-04-25T09:08:03.451 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.492 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1b6403e6:::benchmark_data_smithi088_462306_object33:head# 2024-04-25T09:08:03.492 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.534 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1bce90c6:::benchmark_data_smithi088_462306_object217:head# 2024-04-25T09:08:03.534 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.576 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1cb6ac0c:::benchmark_data_smithi088_462306_object27:head# 2024-04-25T09:08:03.576 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.617 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1cc79080:::benchmark_data_smithi088_462306_object323:head# 2024-04-25T09:08:03.618 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.659 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1d5218b3:::benchmark_data_smithi088_462306_object196:head# 2024-04-25T09:08:03.660 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.693 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1e94d3aa:::benchmark_data_smithi088_462306_object111:head# 2024-04-25T09:08:03.693 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.726 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1eddebf7:::benchmark_data_smithi088_462306_object264:head# 2024-04-25T09:08:03.726 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.760 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1f00585f:::benchmark_data_smithi088_462306_object236:head# 2024-04-25T09:08:03.760 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.793 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20808529:::benchmark_data_smithi088_462306_object151:head# 2024-04-25T09:08:03.793 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.827 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20bdd9c8:::benchmark_data_smithi088_462306_object235:head# 2024-04-25T09:08:03.827 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.860 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20edab27:::benchmark_data_smithi088_462306_object164:head# 2024-04-25T09:08:03.860 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.907 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2160d1fc:::benchmark_data_smithi088_462306_object320:head# 2024-04-25T09:08:03.907 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:03.961 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:220a8d5a:::benchmark_data_smithi088_462306_object77:head# 2024-04-25T09:08:03.961 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.011 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:231dff78:::benchmark_data_smithi088_462306_object239:head# 2024-04-25T09:08:04.011 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.044 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2330fe3c:::benchmark_data_smithi088_462306_object215:head# 2024-04-25T09:08:04.044 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.078 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2337b927:::benchmark_data_smithi088_462306_object191:head# 2024-04-25T09:08:04.078 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.111 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:23a73227:::benchmark_data_smithi088_462306_object212:head# 2024-04-25T09:08:04.111 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.145 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2448079b:::benchmark_data_smithi088_462306_object70:head# 2024-04-25T09:08:04.145 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.178 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:27c74952:::benchmark_data_smithi088_462306_object288:head# 2024-04-25T09:08:04.178 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.211 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:283568ed:::benchmark_data_smithi088_462306_object315:head# 2024-04-25T09:08:04.211 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.245 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:29974a65:::benchmark_data_smithi088_462306_object2:head# 2024-04-25T09:08:04.245 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.278 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2bbd0b7d:::benchmark_data_smithi088_462306_object224:head# 2024-04-25T09:08:04.279 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.312 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f341493:::benchmark_data_smithi088_462306_object96:head# 2024-04-25T09:08:04.312 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.345 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f3e4e35:::benchmark_data_smithi088_462306_object94:head# 2024-04-25T09:08:04.345 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.421 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3089f82a:::benchmark_data_smithi088_462306_object270:head# 2024-04-25T09:08:04.421 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.481 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:30bc55cf:::benchmark_data_smithi088_462306_object179:head# 2024-04-25T09:08:04.481 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.580 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:30ee7b76:::benchmark_data_smithi088_462306_object211:head# 2024-04-25T09:08:04.580 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.613 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:324b040a:::benchmark_data_smithi088_462306_object28:head# 2024-04-25T09:08:04.613 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.647 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:33bdd052:::benchmark_data_smithi088_462306_object185:head# 2024-04-25T09:08:04.647 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.680 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:34a362cf:::benchmark_data_smithi088_462306_object120:head# 2024-04-25T09:08:04.680 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.714 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:36ae6861:::benchmark_data_smithi088_462306_object125:head# 2024-04-25T09:08:04.714 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.747 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:37054d0f:::benchmark_data_smithi088_462306_object194:head# 2024-04-25T09:08:04.747 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.780 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:37196b5a:::benchmark_data_smithi088_462306_object32:head# 2024-04-25T09:08:04.781 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.814 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3a0087c7:::benchmark_data_smithi088_462306_object299:head# 2024-04-25T09:08:04.814 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.847 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3bfc2bf9:::benchmark_data_smithi088_462306_object25:head# 2024-04-25T09:08:04.848 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.881 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3c658397:::benchmark_data_smithi088_462306_object114:head# 2024-04-25T09:08:04.881 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.914 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3ca6aa4a:::benchmark_data_smithi088_462306_object292:head# 2024-04-25T09:08:04.915 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.956 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3df4cae2:::benchmark_data_smithi088_462306_object330:head# 2024-04-25T09:08:04.957 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:04.990 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3e25b978:::benchmark_data_smithi088_462306_object250:head# 2024-04-25T09:08:04.990 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.023 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3e918cd5:::benchmark_data_smithi088_462306_object287:head# 2024-04-25T09:08:05.023 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.057 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3f11fd9d:::benchmark_data_smithi088_462306_object127:head# 2024-04-25T09:08:05.057 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.090 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3f85196b:::benchmark_data_smithi088_462306_object135:head# 2024-04-25T09:08:05.090 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.124 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:41e0f85b:::benchmark_data_smithi088_462306_object200:head# 2024-04-25T09:08:05.124 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.157 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:43088259:::benchmark_data_smithi088_462306_object193:head# 2024-04-25T09:08:05.157 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.191 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:452dfc2d:::benchmark_data_smithi088_462306_object144:head# 2024-04-25T09:08:05.191 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.224 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46484682:::benchmark_data_smithi088_462306_object324:head# 2024-04-25T09:08:05.224 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.258 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46c66152:::benchmark_data_smithi088_462306_object149:head# 2024-04-25T09:08:05.258 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.291 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46f14c22:::benchmark_data_smithi088_462306_object300:head# 2024-04-25T09:08:05.291 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.324 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4b72756f:::benchmark_data_smithi088_462306_object329:head# 2024-04-25T09:08:05.325 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.358 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4ba5396c:::benchmark_data_smithi088_462306_object71:head# 2024-04-25T09:08:05.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.391 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c34ed72:::benchmark_data_smithi088_462306_object154:head# 2024-04-25T09:08:05.392 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.425 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c3e8c07:::benchmark_data_smithi088_462306_object221:head# 2024-04-25T09:08:05.425 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.458 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c6252f1:::benchmark_data_smithi088_462306_object18:head# 2024-04-25T09:08:05.458 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.491 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4cf34651:::benchmark_data_smithi088_462306_object188:head# 2024-04-25T09:08:05.492 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.533 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d14555f:::benchmark_data_smithi088_462306_object121:head# 2024-04-25T09:08:05.534 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.567 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d5ba4b9:::benchmark_data_smithi088_462306_object158:head# 2024-04-25T09:08:05.567 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.617 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d7ce59d:::benchmark_data_smithi088_462306_object104:head# 2024-04-25T09:08:05.617 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.650 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e3dbe21:::benchmark_data_smithi088_462306_object50:head# 2024-04-25T09:08:05.650 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.684 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e4657de:::benchmark_data_smithi088_462306_object73:head# 2024-04-25T09:08:05.684 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.717 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e9de313:::benchmark_data_smithi088_462306_object93:head# 2024-04-25T09:08:05.718 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.751 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:50f6d9da:::benchmark_data_smithi088_462306_object307:head# 2024-04-25T09:08:05.751 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.784 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:52f10993:::benchmark_data_smithi088_462306_object205:head# 2024-04-25T09:08:05.784 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.818 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:539a6d00:::benchmark_data_smithi088_462306_object107:head# 2024-04-25T09:08:05.818 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.853 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:549c3943:::benchmark_data_smithi088_462306_object23:head# 2024-04-25T09:08:05.854 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.906 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:55901106:::benchmark_data_smithi088_462306_object89:head# 2024-04-25T09:08:05.906 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:05.968 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:56f25dc9:::benchmark_data_smithi088_462306_object68:head# 2024-04-25T09:08:05.969 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.019 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:56f2d5c4:::benchmark_data_smithi088_462306_object274:head# 2024-04-25T09:08:06.019 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.060 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:58ffc1f4:::benchmark_data_smithi088_462306_object124:head# 2024-04-25T09:08:06.060 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.102 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:59107f03:::benchmark_data_smithi088_462306_object157:head# 2024-04-25T09:08:06.102 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.136 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5933a9b4:::benchmark_data_smithi088_462306_object4:head# 2024-04-25T09:08:06.136 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.169 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:594172d7:::benchmark_data_smithi088_462306_object95:head# 2024-04-25T09:08:06.169 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.203 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5abcca99:::benchmark_data_smithi088_462306_object85:head# 2024-04-25T09:08:06.203 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.236 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5b39136e:::benchmark_data_smithi088_462306_object168:head# 2024-04-25T09:08:06.236 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.269 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5bbb1972:::benchmark_data_smithi088_462306_object113:head# 2024-04-25T09:08:06.269 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.303 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c0a9a1b:::benchmark_data_smithi088_462306_object80:head# 2024-04-25T09:08:06.303 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.336 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c15d000:::benchmark_data_smithi088_462306_object318:head# 2024-04-25T09:08:06.337 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.381 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c895488:::benchmark_data_smithi088_462306_object58:head# 2024-04-25T09:08:06.381 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.480 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5cffb09b:::benchmark_data_smithi088_462306_object31:head# 2024-04-25T09:08:06.481 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.546 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5daa5792:::benchmark_data_smithi088_462306_object192:head# 2024-04-25T09:08:06.546 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.579 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5e669d4b:::benchmark_data_smithi088_462306_object248:head# 2024-04-25T09:08:06.579 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.613 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5e753d70:::benchmark_data_smithi088_462306_object88:head# 2024-04-25T09:08:06.613 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.646 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5fc0d91a:::benchmark_data_smithi088_462306_object83:head# 2024-04-25T09:08:06.646 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.680 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:60c47461:::benchmark_data_smithi088_462306_object201:head# 2024-04-25T09:08:06.680 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.713 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62524425:::benchmark_data_smithi088_462306_object284:head# 2024-04-25T09:08:06.713 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.746 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62e454a3:::benchmark_data_smithi088_462306_object10:head# 2024-04-25T09:08:06.746 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.780 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62e744cb:::benchmark_data_smithi088_462306_object297:head# 2024-04-25T09:08:06.780 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.813 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:63a33891:::benchmark_data_smithi088_462306_object102:head# 2024-04-25T09:08:06.814 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.847 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:64acde59:::benchmark_data_smithi088_462306_object321:head# 2024-04-25T09:08:06.847 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.914 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:64ca7391:::benchmark_data_smithi088_462306_object213:head# 2024-04-25T09:08:06.914 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:06.947 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:65bc1817:::benchmark_data_smithi088_462306_object303:head# 2024-04-25T09:08:06.947 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.014 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6633f92d:::benchmark_data_smithi088_462306_object126:head# 2024-04-25T09:08:07.014 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.047 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:678dbd08:::benchmark_data_smithi088_462306_object52:head# 2024-04-25T09:08:07.048 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.089 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67a6186b:::benchmark_data_smithi088_462306_object327:head# 2024-04-25T09:08:07.090 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.123 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67aa77b1:::benchmark_data_smithi088_462306_object44:head# 2024-04-25T09:08:07.123 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.156 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6806c11f:::benchmark_data_smithi088_462306_object261:head# 2024-04-25T09:08:07.157 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.190 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:68b0f3d0:::benchmark_data_smithi088_462306_object328:head# 2024-04-25T09:08:07.190 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.224 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6a010110:::benchmark_data_smithi088_462306_object296:head# 2024-04-25T09:08:07.224 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.257 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6b5538cb:::benchmark_data_smithi088_462306_object308:head# 2024-04-25T09:08:07.257 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.290 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6b7b59b0:::benchmark_data_smithi088_462306_object218:head# 2024-04-25T09:08:07.290 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.324 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6bbd025e:::benchmark_data_smithi088_462306_object141:head# 2024-04-25T09:08:07.324 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.357 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6c16e74e:::benchmark_data_smithi088_462306_object241:head# 2024-04-25T09:08:07.357 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.390 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6d493727:::benchmark_data_smithi088_462306_object86:head# 2024-04-25T09:08:07.391 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.424 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7041d2eb:::benchmark_data_smithi088_462306_object171:head# 2024-04-25T09:08:07.424 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.457 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:72ab0865:::benchmark_data_smithi088_462306_object42:head# 2024-04-25T09:08:07.458 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.491 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:73e18593:::benchmark_data_smithi088_462306_object234:head# 2024-04-25T09:08:07.491 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.524 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7411cf6e:::benchmark_data_smithi088_462306_object305:head# 2024-04-25T09:08:07.524 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.558 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:749c107b:::benchmark_data_smithi088_462306_object29:head# 2024-04-25T09:08:07.558 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.591 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7759d088:::benchmark_data_smithi088_462306_object310:head# 2024-04-25T09:08:07.591 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.625 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:777330ea:::benchmark_data_smithi088_462306_object262:head# 2024-04-25T09:08:07.625 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.658 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:77982bca:::benchmark_data_smithi088_462306_object269:head# 2024-04-25T09:08:07.658 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.692 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7879b0d4:::benchmark_data_smithi088_462306_object313:head# 2024-04-25T09:08:07.692 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.725 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7c39dbd4:::benchmark_data_smithi088_462306_object45:head# 2024-04-25T09:08:07.725 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.775 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7cb590f7:::benchmark_data_smithi088_462306_object177:head# 2024-04-25T09:08:07.775 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.809 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7d29cf47:::benchmark_data_smithi088_462306_object249:head# 2024-04-25T09:08:07.809 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.842 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7db41ef9:::benchmark_data_smithi088_462306_object190:head# 2024-04-25T09:08:07.842 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.876 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7de60352:::benchmark_data_smithi088_462306_object210:head# 2024-04-25T09:08:07.876 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:07.951 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7e59f398:::benchmark_data_smithi088_462306_object110:head# 2024-04-25T09:08:07.951 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:08.001 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7efa7304:::benchmark_data_smithi088_462306_object74:head# 2024-04-25T09:08:08.001 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:08.035 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7fd54788:::benchmark_data_smithi088_462306_object133:head# 2024-04-25T09:08:08.035 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:08:08.068 INFO:tasks.workunit.client.0.smithi088.stdout:write_pg epoch 8 info 1.0( v 8'158 (0'0,8'158] local-lis/les=6/7 n=158 ec=6/6 lis/c=6/6 les/c/f=7/7/0 sis=6) 2024-04-25T09:08:08.110 INFO:tasks.workunit.client.0.smithi088.stdout:Import successful 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:115: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:08:08.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:08:08.651 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:08:08.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:08:08.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:08:08.652 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:08:08.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:08:08.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:08:08.654 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:08:08.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:08:08.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:08:08.654 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:08:08.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:08:08.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:08:08.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:08:08.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:08:08.689 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:08:08.699+0000 7f8f7ca327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:08.692 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:08:08.703+0000 7f8f7ca327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:08.693 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:08:08.703+0000 7f8f7ca327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:09.069 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:08:09.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:08:09.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:08:09.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:08:09.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:08:09.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:09.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:08:09.070 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:08:09.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:09.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:09.240 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:08:09.251+0000 7f8f7ca327c0 -1 Falling back to public interface 2024-04-25T09:08:09.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:10.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:10.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:10.551 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:08:10.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:08:10.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:10.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:10.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:11.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:11.983 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:08:11.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:11.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:08:11.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:11.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:12.253 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:08:12.263+0000 7f8f7ca327c0 -1 osd.0 19 log_to_monitors true 2024-04-25T09:08:12.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:13.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:13.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:13.411 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:08:13.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:08:13.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:13.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:08:13.829 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 22 up_thru 22 down_at 20 last_clean_interval [11,19) [v2:127.0.0.1:6802/2297734166,v1:127.0.0.1:6803/2297734166] [v2:127.0.0.1:6804/2297734166,v1:127.0.0.1:6805/2297734166] exists,up 168d882f-4939-4337-8b16-26a6cead1b6d 2024-04-25T09:08:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:08:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:08:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:117: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:08:13.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:08:13.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:08:13.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:08:13.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:08:13.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:08:13.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:08:13.934 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:08:14.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:08:14.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:08:14.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:08:14.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:08:14.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=94489280515 2024-04-25T09:08:14.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 94489280515 2024-04-25T09:08:14.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-94489280515' 2024-04-25T09:08:14.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:08:14.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-94489280515 2024-04-25T09:08:14.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:08:14.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:08:14.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-94489280515 2024-04-25T09:08:14.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:08:14.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=94489280515 2024-04-25T09:08:14.607 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 94489280515 2024-04-25T09:08:14.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 94489280515' 2024-04-25T09:08:14.608 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:08:15.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 94489280515 2024-04-25T09:08:15.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:08:16.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:08:16.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:08:16.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 94489280515 -lt 94489280515 2024-04-25T09:08:16.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:08:16.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:08:16.424 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:08:17.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:08:17.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:08:17.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:08:17.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:08:17.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:08:17.042 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:08:17.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:08:17.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:08:17.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:08:17.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:08:17.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:08:17.439 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:08:17.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:08:17.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:08:17.983 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:08:17.984 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: seq 1 50 2024-04-25T09:08:17.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:17.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:18.608 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:18.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:19.453 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:19.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:19.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:20.231 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:20.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:21.101 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:21.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:21.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:21.906 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:21.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:22.717 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:22.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:22.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:23.445 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:23.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:24.215 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:24.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:24.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:24.877 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:24.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:25.590 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:25.612 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:25.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:26.302 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:26.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:26.996 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:27.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:27.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:27.901 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:27.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:28.578 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:28.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:28.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:29.315 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:29.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:30.177 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:30.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:30.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:30.930 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:30.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:31.658 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:31.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:31.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:32.444 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:32.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:33.206 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:33.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:33.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:33.959 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:33.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:34.671 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:34.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:34.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:35.391 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:35.411 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:36.161 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:36.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:36.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:37.032 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:37.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:37.742 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:37.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:37.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:38.487 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:38.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:39.240 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:39.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:39.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:39.876 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:39.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:40.629 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:40.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:40.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:41.358 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:41.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:42.254 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:42.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:42.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:43.099 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:43.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:43.961 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:43.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:43.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:44.613 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:44.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:45.433 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:45.451 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:45.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:46.195 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:46.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:47.006 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:47.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:47.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:47.835 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:47.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:48.512 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:48.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:48.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:49.316 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:49.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:50.118 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:50.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:50.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:50.897 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:50.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:51.834 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:51.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:51.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:52.511 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:52.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:53.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:53.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:54.085 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:54.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:54.729 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:54.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:54.746 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:55.524 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:55.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:56.302 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:56.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:56.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:57.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:57.859 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:57.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:57.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:08:58.670 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:08:58.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:08:59.457 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:08:59.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:08:59.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:00.352 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:00.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:01.122 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:01.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:01.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:01.976 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:01.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:02.788 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:02.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:02.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:03.591 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:03.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:04.345 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:04.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:04.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:05.139 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:05.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:05.901 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:05.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:05.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:06.696 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:06.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:07.516 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:07.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:07.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:08.294 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:08.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:08.996 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:09.016 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:09.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:09.816 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:09.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:10.511 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:10.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:10.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:11.314 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:11.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:12.117 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:12.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:12.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:12.921 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:12.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:13.765 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:13.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:13.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:14.519 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:14.536 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:15.355 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:15.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:15.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:16.133 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:16.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:16.895 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:16.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:16.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:17.740 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:17.761 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:18.476 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:18.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:19.304 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:19.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:20.116 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:20.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:20.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:20.877 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:20.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:21.681 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:21.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:21.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:22.668 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:22.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:23.522 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:23.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:23.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:24.258 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:24.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:25.030 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:25.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:25.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:25.800 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:25.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:26.654 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:26.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:26.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:27.558 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:27.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:28.285 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:28.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:28.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:29.197 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:29.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:29.999 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:30.025 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:30.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:30.802 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:30.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:31.572 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:31.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:31.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:32.334 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:32.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:33.154 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:33.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:33.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:33.781 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:33.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:34.585 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:34.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:09:34.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:35.329 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:35.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:36.116 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:36.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:127: TEST_import_after_merge_and_gap: rados -p foo bench 1 write -b 1024 --no-cleanup 2024-04-25T09:09:36.244 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T09:09:36.244 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 1 seconds or 0 objects 2024-04-25T09:09:36.244 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_467817 2024-04-25T09:09:36.244 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:09:36.244 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:09:37.244 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 128 112 0.109361 0.109375 0.145345 0.134118 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 1.09546 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 128 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 1024 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 1024 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.114108 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.109375 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.109375 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 116 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 0 2024-04-25T09:09:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 112 2024-04-25T09:09:38.246 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 112 2024-04-25T09:09:38.246 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.13635 2024-04-25T09:09:38.246 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0351269 2024-04-25T09:09:38.246 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.201522 2024-04-25T09:09:38.246 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.0306323 2024-04-25T09:09:38.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:128: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:09:38.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:09:38.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:09:38.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:09:38.286 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:09:38.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:09:38.288 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:09:38.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:09:38.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:09:38.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:09:38.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:09:38.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:09:38.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:09:38.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:09:38.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:09:38.410 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:09:38.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:09:38.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:09:38.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:38.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:09:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=94489280534 2024-04-25T09:09:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 94489280534 2024-04-25T09:09:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-94489280534' 2024-04-25T09:09:39.008 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:39.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-94489280534 2024-04-25T09:09:39.008 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:39.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:09:39.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-94489280534 2024-04-25T09:09:39.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:39.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=94489280534 2024-04-25T09:09:39.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 94489280534' 2024-04-25T09:09:39.012 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 94489280534 2024-04-25T09:09:39.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:39.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 94489280532 -lt 94489280534 2024-04-25T09:09:39.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:09:40.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:09:40.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:40.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 94489280534 -lt 94489280534 2024-04-25T09:09:40.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:09:40.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:40.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:41.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:09:41.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:09:41.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:09:41.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:09:41.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:09:41.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:09:41.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:09:41.421 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:09:41.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:09:41.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:09:41.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:41.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:42.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:09:42.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:09:42.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:09:42.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:129: TEST_import_after_merge_and_gap: ceph tell osd.0 send_beacon 2024-04-25T09:09:42.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:130: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:09:47.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:131: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:09:48.367 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is set 2024-04-25T09:09:48.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:132: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:09:49.137 INFO:tasks.workunit.client.0.smithi088.stderr:nodown is unset 2024-04-25T09:09:49.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:133: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:09:54.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:135: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:09:54.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:09:54.160 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:09:54.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:09:54.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:09:54.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:09:54.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:09:54.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:138: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:09:55.529 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e227 pg_nums {1={18=1}} deleted_pools ) 2024-04-25T09:09:55.529 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.1 2024-04-25T09:09:55.530 INFO:tasks.workunit.client.0.smithi088.stderr:PG export's map 8 is older than OSD's oldest_map 61 2024-04-25T09:09:55.530 INFO:tasks.workunit.client.0.smithi088.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:09:56.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:140: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export-remove --pgid 1.0 --force --file td/pg-split-merge/1.0.later 2024-04-25T09:09:56.842 INFO:tasks.workunit.client.0.smithi088.stderr:Exporting 1.0 info 1.0( v 223'310 (8'181,223'310] local-lis/les=22/23 n=467 ec=6/6 lis/c=22/22 les/c/f=23/23/0 sis=22) 2024-04-25T09:09:56.861 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:004ab2c0:::benchmark_data_smithi088_467817_object117:head# 2024-04-25T09:09:56.861 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0054a4c5:::benchmark_data_smithi088_462306_object56:head# 2024-04-25T09:09:56.862 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0091abb9:::benchmark_data_smithi088_462306_object216:head# 2024-04-25T09:09:56.863 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:01a57ad0:::benchmark_data_smithi088_462306_object317:head# 2024-04-25T09:09:56.863 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:01e683c2:::benchmark_data_smithi088_462306_object106:head# 2024-04-25T09:09:56.864 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0206297e:::benchmark_data_smithi088_467817_object18:head# 2024-04-25T09:09:56.865 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:021a8ccb:::benchmark_data_smithi088_462306_object108:head# 2024-04-25T09:09:56.865 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:022d1a7b:::benchmark_data_smithi088_462306_object223:head# 2024-04-25T09:09:56.866 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0265fcfd:::benchmark_data_smithi088_467817_object94:head# 2024-04-25T09:09:56.866 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0496a541:::benchmark_data_smithi088_462306_object115:head# 2024-04-25T09:09:56.867 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:052189ea:::benchmark_data_smithi088_462306_object198:head# 2024-04-25T09:09:56.867 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:052d5fd4:::benchmark_data_smithi088_462306_object14:head# 2024-04-25T09:09:56.868 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0589e269:::benchmark_data_smithi088_467817_object98:head# 2024-04-25T09:09:56.868 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:05f37ee5:::benchmark_data_smithi088_467817_object19:head# 2024-04-25T09:09:56.869 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:06462b6c:::benchmark_data_smithi088_462306_object237:head# 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0665cd86:::benchmark_data_smithi088_462306_object266:head# 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:072ee852:::benchmark_data_smithi088_467817_object116:head# 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:089ed0f9:::benchmark_data_smithi088_467817_object47:head# 2024-04-25T09:09:56.871 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:08ef67f6:::benchmark_data_smithi088_462306_object167:head# 2024-04-25T09:09:56.872 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:09033334:::benchmark_data_smithi088_462306_object57:head# 2024-04-25T09:09:56.872 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:09cc47f1:::benchmark_data_smithi088_467817_object90:head# 2024-04-25T09:09:56.873 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0a07ee66:::benchmark_data_smithi088_462306_object139:head# 2024-04-25T09:09:56.874 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0a9dbfd3:::benchmark_data_smithi088_462306_object147:head# 2024-04-25T09:09:56.874 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b31df74:::benchmark_data_smithi088_462306_object103:head# 2024-04-25T09:09:56.874 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b343d83:::benchmark_data_smithi088_467817_object8:head# 2024-04-25T09:09:56.875 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b35794e:::benchmark_data_smithi088_462306_object309:head# 2024-04-25T09:09:56.879 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b54d91a:::benchmark_data_smithi088_462306_object143:head# 2024-04-25T09:09:56.880 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0b783152:::benchmark_data_smithi088_462306_object53:head# 2024-04-25T09:09:56.880 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0be5222f:::benchmark_data_smithi088_462306_object295:head# 2024-04-25T09:09:56.881 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0c2e7645:::benchmark_data_smithi088_462306_object65:head# 2024-04-25T09:09:56.881 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0d3bf401:::benchmark_data_smithi088_462306_object84:head# 2024-04-25T09:09:56.882 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0da5c762:::benchmark_data_smithi088_462306_object137:head# 2024-04-25T09:09:56.882 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0efc8997:::benchmark_data_smithi088_462306_object228:head# 2024-04-25T09:09:56.883 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0f3cc8d1:::benchmark_data_smithi088_462306_object105:head# 2024-04-25T09:09:56.883 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0f6ddd44:::benchmark_data_smithi088_462306_object131:head# 2024-04-25T09:09:56.883 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1039c45c:::benchmark_data_smithi088_467817_object33:head# 2024-04-25T09:09:56.884 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:10ea7e0c:::benchmark_data_smithi088_467817_object31:head# 2024-04-25T09:09:56.885 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:11a8d835:::benchmark_data_smithi088_462306_object20:head# 2024-04-25T09:09:56.885 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:12bd0b48:::benchmark_data_smithi088_467817_object83:head# 2024-04-25T09:09:56.886 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:12bf1666:::benchmark_data_smithi088_462306_object63:head# 2024-04-25T09:09:56.886 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1368f5d1:::benchmark_data_smithi088_462306_object208:head# 2024-04-25T09:09:56.887 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:13b33233:::benchmark_data_smithi088_462306_object40:head# 2024-04-25T09:09:56.887 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:13c66d49:::benchmark_data_smithi088_467817_object46:head# 2024-04-25T09:09:56.888 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:140e1661:::benchmark_data_smithi088_462306_object247:head# 2024-04-25T09:09:56.888 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:148e010e:::benchmark_data_smithi088_467817_object25:head# 2024-04-25T09:09:56.889 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:151d580a:::benchmark_data_smithi088_462306_object59:head# 2024-04-25T09:09:56.889 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:15cce5ab:::benchmark_data_smithi088_467817_object126:head# 2024-04-25T09:09:56.890 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:16cc41f5:::benchmark_data_smithi088_462306_object67:head# 2024-04-25T09:09:56.890 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:17a2c91b:::benchmark_data_smithi088_462306_object17:head# 2024-04-25T09:09:56.891 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:17f6836e:::benchmark_data_smithi088_462306_object256:head# 2024-04-25T09:09:56.891 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:186d6a7e:::benchmark_data_smithi088_462306_object0:head# 2024-04-25T09:09:56.892 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:195aac3a:::benchmark_data_smithi088_467817_object88:head# 2024-04-25T09:09:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:19967450:::benchmark_data_smithi088_462306_object123:head# 2024-04-25T09:09:56.893 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1a55af78:::benchmark_data_smithi088_462306_object75:head# 2024-04-25T09:09:56.894 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1a90b07a:::benchmark_data_smithi088_467817_object40:head# 2024-04-25T09:09:56.894 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1acd4c2d:::benchmark_data_smithi088_462306_object187:head# 2024-04-25T09:09:56.895 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1b6403e6:::benchmark_data_smithi088_462306_object33:head# 2024-04-25T09:09:56.895 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1bce90c6:::benchmark_data_smithi088_462306_object217:head# 2024-04-25T09:09:56.896 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1cb6ac0c:::benchmark_data_smithi088_462306_object27:head# 2024-04-25T09:09:56.897 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1cc79080:::benchmark_data_smithi088_462306_object323:head# 2024-04-25T09:09:56.897 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1d5218b3:::benchmark_data_smithi088_462306_object196:head# 2024-04-25T09:09:56.898 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1e8c7435:::benchmark_data_smithi088_467817_object48:head# 2024-04-25T09:09:56.899 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1e94d3aa:::benchmark_data_smithi088_462306_object111:head# 2024-04-25T09:09:56.900 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1eddebf7:::benchmark_data_smithi088_462306_object264:head# 2024-04-25T09:09:56.900 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1f00585f:::benchmark_data_smithi088_462306_object236:head# 2024-04-25T09:09:56.901 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1f02ec56:::benchmark_data_smithi088_467817_object89:head# 2024-04-25T09:09:56.901 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1fd6b57c:::benchmark_data_smithi088_467817_object62:head# 2024-04-25T09:09:56.902 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:20808529:::benchmark_data_smithi088_462306_object151:head# 2024-04-25T09:09:56.902 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:20bdd9c8:::benchmark_data_smithi088_462306_object235:head# 2024-04-25T09:09:56.903 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:20edab27:::benchmark_data_smithi088_462306_object164:head# 2024-04-25T09:09:56.904 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:214dd915:::benchmark_data_smithi088_467817_object85:head# 2024-04-25T09:09:56.904 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2160d1fc:::benchmark_data_smithi088_462306_object320:head# 2024-04-25T09:09:56.905 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:220a8d5a:::benchmark_data_smithi088_462306_object77:head# 2024-04-25T09:09:56.905 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:225cea0d:::benchmark_data_smithi088_467817_object75:head# 2024-04-25T09:09:56.906 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:22cdb4aa:::benchmark_data_smithi088_467817_object84:head# 2024-04-25T09:09:56.906 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:231dff78:::benchmark_data_smithi088_462306_object239:head# 2024-04-25T09:09:56.907 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2330fe3c:::benchmark_data_smithi088_462306_object215:head# 2024-04-25T09:09:56.907 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2337b927:::benchmark_data_smithi088_462306_object191:head# 2024-04-25T09:09:56.908 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:23a73227:::benchmark_data_smithi088_462306_object212:head# 2024-04-25T09:09:56.909 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:24024e88:::benchmark_data_smithi088_467817_object101:head# 2024-04-25T09:09:56.909 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2448079b:::benchmark_data_smithi088_462306_object70:head# 2024-04-25T09:09:56.910 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:24d502c0:::benchmark_data_smithi088_467817_object99:head# 2024-04-25T09:09:56.910 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:258537a5:::benchmark_data_smithi088_467817_object112:head# 2024-04-25T09:09:56.911 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:27359498:::benchmark_data_smithi088_467817_object96:head# 2024-04-25T09:09:56.911 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:27c74952:::benchmark_data_smithi088_462306_object288:head# 2024-04-25T09:09:56.912 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:283568ed:::benchmark_data_smithi088_462306_object315:head# 2024-04-25T09:09:56.912 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:28fdadb1:::benchmark_data_smithi088_467817_object54:head# 2024-04-25T09:09:56.913 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:28ff6e11:::benchmark_data_smithi088_467817_object27:head# 2024-04-25T09:09:56.913 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:29974a65:::benchmark_data_smithi088_462306_object2:head# 2024-04-25T09:09:56.914 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2bbd0b7d:::benchmark_data_smithi088_462306_object224:head# 2024-04-25T09:09:56.914 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2e92b507:::benchmark_data_smithi088_467817_object37:head# 2024-04-25T09:09:56.915 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f341493:::benchmark_data_smithi088_462306_object96:head# 2024-04-25T09:09:56.915 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f3e4e35:::benchmark_data_smithi088_462306_object94:head# 2024-04-25T09:09:56.915 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3089f82a:::benchmark_data_smithi088_462306_object270:head# 2024-04-25T09:09:56.916 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:30bc55cf:::benchmark_data_smithi088_462306_object179:head# 2024-04-25T09:09:56.916 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:30ee7b76:::benchmark_data_smithi088_462306_object211:head# 2024-04-25T09:09:56.917 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:324b040a:::benchmark_data_smithi088_462306_object28:head# 2024-04-25T09:09:56.917 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:325722a7:::benchmark_data_smithi088_467817_object124:head# 2024-04-25T09:09:56.919 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:33bdd052:::benchmark_data_smithi088_462306_object185:head# 2024-04-25T09:09:56.919 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:34099f88:::benchmark_data_smithi088_467817_object41:head# 2024-04-25T09:09:56.920 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:34a362cf:::benchmark_data_smithi088_462306_object120:head# 2024-04-25T09:09:56.920 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:353eace7:::benchmark_data_smithi088_467817_object106:head# 2024-04-25T09:09:56.921 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3577de39:::benchmark_data_smithi088_467817_object29:head# 2024-04-25T09:09:56.921 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:36560ed1:::benchmark_data_smithi088_467817_object67:head# 2024-04-25T09:09:56.921 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:36ae6861:::benchmark_data_smithi088_462306_object125:head# 2024-04-25T09:09:56.922 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:36f56c74:::benchmark_data_smithi088_467817_object60:head# 2024-04-25T09:09:56.922 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:37054d0f:::benchmark_data_smithi088_462306_object194:head# 2024-04-25T09:09:56.923 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:37196b5a:::benchmark_data_smithi088_462306_object32:head# 2024-04-25T09:09:56.923 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3842e360:::benchmark_data_smithi088_467817_object50:head# 2024-04-25T09:09:56.924 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3a0087c7:::benchmark_data_smithi088_462306_object299:head# 2024-04-25T09:09:56.924 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3bfc2bf9:::benchmark_data_smithi088_462306_object25:head# 2024-04-25T09:09:56.925 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3c658397:::benchmark_data_smithi088_462306_object114:head# 2024-04-25T09:09:56.925 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3ca6aa4a:::benchmark_data_smithi088_462306_object292:head# 2024-04-25T09:09:56.926 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3dab0fd5:::benchmark_data_smithi088_467817_object108:head# 2024-04-25T09:09:56.926 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3db8775a:::benchmark_data_smithi088_467817_object55:head# 2024-04-25T09:09:56.927 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3df4cae2:::benchmark_data_smithi088_462306_object330:head# 2024-04-25T09:09:56.927 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3e25b978:::benchmark_data_smithi088_462306_object250:head# 2024-04-25T09:09:56.928 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3e918cd5:::benchmark_data_smithi088_462306_object287:head# 2024-04-25T09:09:56.928 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3f11fd9d:::benchmark_data_smithi088_462306_object127:head# 2024-04-25T09:09:56.929 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3f85196b:::benchmark_data_smithi088_462306_object135:head# 2024-04-25T09:09:56.929 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:41e0f85b:::benchmark_data_smithi088_462306_object200:head# 2024-04-25T09:09:56.930 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:43088259:::benchmark_data_smithi088_462306_object193:head# 2024-04-25T09:09:56.930 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4359c343:::benchmark_data_smithi088_467817_object76:head# 2024-04-25T09:09:56.931 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:452dfc2d:::benchmark_data_smithi088_462306_object144:head# 2024-04-25T09:09:56.931 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:46484682:::benchmark_data_smithi088_462306_object324:head# 2024-04-25T09:09:56.932 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:46c66152:::benchmark_data_smithi088_462306_object149:head# 2024-04-25T09:09:56.932 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:46f14c22:::benchmark_data_smithi088_462306_object300:head# 2024-04-25T09:09:56.933 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:496c1623:::benchmark_data_smithi088_467817_object22:head# 2024-04-25T09:09:56.933 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4985fbff:::benchmark_data_smithi088_467817_object68:head# 2024-04-25T09:09:56.934 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:49e3049a:::benchmark_data_smithi088_467817_object79:head# 2024-04-25T09:09:56.934 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4b72756f:::benchmark_data_smithi088_462306_object329:head# 2024-04-25T09:09:56.935 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4ba5396c:::benchmark_data_smithi088_462306_object71:head# 2024-04-25T09:09:56.935 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4c34ed72:::benchmark_data_smithi088_462306_object154:head# 2024-04-25T09:09:56.936 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4c3e8c07:::benchmark_data_smithi088_462306_object221:head# 2024-04-25T09:09:56.936 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4c6252f1:::benchmark_data_smithi088_462306_object18:head# 2024-04-25T09:09:56.937 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4cf34651:::benchmark_data_smithi088_462306_object188:head# 2024-04-25T09:09:56.937 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4d14555f:::benchmark_data_smithi088_462306_object121:head# 2024-04-25T09:09:56.937 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4d5ba4b9:::benchmark_data_smithi088_462306_object158:head# 2024-04-25T09:09:56.938 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4d7ce59d:::benchmark_data_smithi088_462306_object104:head# 2024-04-25T09:09:56.938 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4e3dbe21:::benchmark_data_smithi088_462306_object50:head# 2024-04-25T09:09:56.938 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4e4657de:::benchmark_data_smithi088_462306_object73:head# 2024-04-25T09:09:56.939 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4e9de313:::benchmark_data_smithi088_462306_object93:head# 2024-04-25T09:09:56.939 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4ecc8c89:::benchmark_data_smithi088_467817_object71:head# 2024-04-25T09:09:56.940 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:502a7648:::benchmark_data_smithi088_467817_object118:head# 2024-04-25T09:09:56.941 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:507e74a0:::benchmark_data_smithi088_467817_object109:head# 2024-04-25T09:09:56.941 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:50f6d9da:::benchmark_data_smithi088_462306_object307:head# 2024-04-25T09:09:56.942 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:52f10993:::benchmark_data_smithi088_462306_object205:head# 2024-04-25T09:09:56.942 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:539a6d00:::benchmark_data_smithi088_462306_object107:head# 2024-04-25T09:09:56.943 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:549c3943:::benchmark_data_smithi088_462306_object23:head# 2024-04-25T09:09:56.943 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:55901106:::benchmark_data_smithi088_462306_object89:head# 2024-04-25T09:09:56.944 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:56f25dc9:::benchmark_data_smithi088_462306_object68:head# 2024-04-25T09:09:56.944 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:56f2d5c4:::benchmark_data_smithi088_462306_object274:head# 2024-04-25T09:09:56.944 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:58ffc1f4:::benchmark_data_smithi088_462306_object124:head# 2024-04-25T09:09:56.945 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:59107f03:::benchmark_data_smithi088_462306_object157:head# 2024-04-25T09:09:56.945 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:59287275:::benchmark_data_smithi088_467817_object17:head# 2024-04-25T09:09:56.946 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5933a9b4:::benchmark_data_smithi088_462306_object4:head# 2024-04-25T09:09:56.947 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:594172d7:::benchmark_data_smithi088_462306_object95:head# 2024-04-25T09:09:56.947 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:59d8cbe5:::benchmark_data_smithi088_467817_object3:head# 2024-04-25T09:09:56.948 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5abcca99:::benchmark_data_smithi088_462306_object85:head# 2024-04-25T09:09:56.948 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5b39136e:::benchmark_data_smithi088_462306_object168:head# 2024-04-25T09:09:56.948 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5bbb1972:::benchmark_data_smithi088_462306_object113:head# 2024-04-25T09:09:56.949 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5c0a9a1b:::benchmark_data_smithi088_462306_object80:head# 2024-04-25T09:09:56.949 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5c15d000:::benchmark_data_smithi088_462306_object318:head# 2024-04-25T09:09:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5c895488:::benchmark_data_smithi088_462306_object58:head# 2024-04-25T09:09:56.950 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5cbbf528:::benchmark_data_smithi088_467817_object127:head# 2024-04-25T09:09:56.952 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5cffb09b:::benchmark_data_smithi088_462306_object31:head# 2024-04-25T09:09:56.952 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5daa5792:::benchmark_data_smithi088_462306_object192:head# 2024-04-25T09:09:56.953 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5e2ada91:::benchmark_data_smithi088_467817_object32:head# 2024-04-25T09:09:56.953 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5e669d4b:::benchmark_data_smithi088_462306_object248:head# 2024-04-25T09:09:56.954 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5e753d70:::benchmark_data_smithi088_462306_object88:head# 2024-04-25T09:09:56.954 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5f00c71b:::benchmark_data_smithi088_467817_object7:head# 2024-04-25T09:09:56.954 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5fc0d91a:::benchmark_data_smithi088_462306_object83:head# 2024-04-25T09:09:56.955 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:60c47461:::benchmark_data_smithi088_462306_object201:head# 2024-04-25T09:09:56.955 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6116e046:::benchmark_data_smithi088_467817_object87:head# 2024-04-25T09:09:56.956 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:62524425:::benchmark_data_smithi088_462306_object284:head# 2024-04-25T09:09:56.957 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6299f2a8:::benchmark_data_smithi088_467817_object97:head# 2024-04-25T09:09:56.957 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:62e454a3:::benchmark_data_smithi088_462306_object10:head# 2024-04-25T09:09:56.957 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:62e744cb:::benchmark_data_smithi088_462306_object297:head# 2024-04-25T09:09:56.958 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:637a868d:::benchmark_data_smithi088_467817_object0:head# 2024-04-25T09:09:56.958 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:63a33891:::benchmark_data_smithi088_462306_object102:head# 2024-04-25T09:09:56.959 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:64acde59:::benchmark_data_smithi088_462306_object321:head# 2024-04-25T09:09:56.959 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:64ca7391:::benchmark_data_smithi088_462306_object213:head# 2024-04-25T09:09:56.960 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:65bc1817:::benchmark_data_smithi088_462306_object303:head# 2024-04-25T09:09:56.960 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6633f92d:::benchmark_data_smithi088_462306_object126:head# 2024-04-25T09:09:56.961 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:678dbd08:::benchmark_data_smithi088_462306_object52:head# 2024-04-25T09:09:56.961 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:67a6186b:::benchmark_data_smithi088_462306_object327:head# 2024-04-25T09:09:56.962 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:67aa77b1:::benchmark_data_smithi088_462306_object44:head# 2024-04-25T09:09:56.962 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:67befc61:::benchmark_data_smithi088_467817_object57:head# 2024-04-25T09:09:56.963 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6806c11f:::benchmark_data_smithi088_462306_object261:head# 2024-04-25T09:09:56.964 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:68b0f3d0:::benchmark_data_smithi088_462306_object328:head# 2024-04-25T09:09:56.964 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6a010110:::benchmark_data_smithi088_462306_object296:head# 2024-04-25T09:09:56.965 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6a52debd:::benchmark_data_smithi088_467817_object107:head# 2024-04-25T09:09:56.965 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6b5538cb:::benchmark_data_smithi088_462306_object308:head# 2024-04-25T09:09:56.966 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6b7b59b0:::benchmark_data_smithi088_462306_object218:head# 2024-04-25T09:09:56.966 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6bbd025e:::benchmark_data_smithi088_462306_object141:head# 2024-04-25T09:09:56.967 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6c16e74e:::benchmark_data_smithi088_462306_object241:head# 2024-04-25T09:09:56.967 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6d493727:::benchmark_data_smithi088_462306_object86:head# 2024-04-25T09:09:56.968 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6ec178d7:::benchmark_data_smithi088_467817_object4:head# 2024-04-25T09:09:56.969 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6f1ee242:::benchmark_data_smithi088_467817_object73:head# 2024-04-25T09:09:56.969 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7041d2eb:::benchmark_data_smithi088_462306_object171:head# 2024-04-25T09:09:56.970 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:70889470:::benchmark_data_smithi088_467817_object82:head# 2024-04-25T09:09:56.970 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:70c95171:::benchmark_data_smithi088_467817_object20:head# 2024-04-25T09:09:56.971 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:71511fb6:::benchmark_data_smithi088_467817_object52:head# 2024-04-25T09:09:56.971 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7260b26c:::benchmark_data_smithi088_467817_object6:head# 2024-04-25T09:09:56.972 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:72ab0865:::benchmark_data_smithi088_462306_object42:head# 2024-04-25T09:09:56.973 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:73e18593:::benchmark_data_smithi088_462306_object234:head# 2024-04-25T09:09:56.974 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7411cf6e:::benchmark_data_smithi088_462306_object305:head# 2024-04-25T09:09:56.974 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:749c107b:::benchmark_data_smithi088_462306_object29:head# 2024-04-25T09:09:56.975 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7759d088:::benchmark_data_smithi088_462306_object310:head# 2024-04-25T09:09:56.975 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:777330ea:::benchmark_data_smithi088_462306_object262:head# 2024-04-25T09:09:56.976 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:77982bca:::benchmark_data_smithi088_462306_object269:head# 2024-04-25T09:09:56.976 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:78711619:::benchmark_data_smithi088_467817_object14:head# 2024-04-25T09:09:56.977 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7879b0d4:::benchmark_data_smithi088_462306_object313:head# 2024-04-25T09:09:56.978 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7c13826a:::benchmark_data_smithi088_467817_object21:head# 2024-04-25T09:09:56.978 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7c39dbd4:::benchmark_data_smithi088_462306_object45:head# 2024-04-25T09:09:56.979 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7cb590f7:::benchmark_data_smithi088_462306_object177:head# 2024-04-25T09:09:56.979 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7d29cf47:::benchmark_data_smithi088_462306_object249:head# 2024-04-25T09:09:56.980 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7d6d5f52:::benchmark_data_smithi088_467817_object115:head# 2024-04-25T09:09:56.981 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7db41ef9:::benchmark_data_smithi088_462306_object190:head# 2024-04-25T09:09:56.982 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7de60352:::benchmark_data_smithi088_462306_object210:head# 2024-04-25T09:09:56.982 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7e59f398:::benchmark_data_smithi088_462306_object110:head# 2024-04-25T09:09:56.983 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7efa7304:::benchmark_data_smithi088_462306_object74:head# 2024-04-25T09:09:56.983 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7fd54788:::benchmark_data_smithi088_462306_object133:head# 2024-04-25T09:09:56.984 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81099aed:::benchmark_data_smithi088_462306_object181:head# 2024-04-25T09:09:56.985 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81ae27f9:::benchmark_data_smithi088_462306_object252:head# 2024-04-25T09:09:56.985 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81b12e8e:::benchmark_data_smithi088_462306_object161:head# 2024-04-25T09:09:56.985 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81b4c706:::benchmark_data_smithi088_467817_object58:head# 2024-04-25T09:09:56.986 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:81e3c822:::benchmark_data_smithi088_462306_object64:head# 2024-04-25T09:09:56.986 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:821c493f:::benchmark_data_smithi088_467817_object119:head# 2024-04-25T09:09:56.987 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:82743540:::benchmark_data_smithi088_462306_object276:head# 2024-04-25T09:09:56.987 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:842cb2a9:::benchmark_data_smithi088_462306_object278:head# 2024-04-25T09:09:56.988 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:84c54114:::benchmark_data_smithi088_462306_object66:head# 2024-04-25T09:09:56.988 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:85078920:::benchmark_data_smithi088_462306_object62:head# 2024-04-25T09:09:56.989 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:85f7c2db:::benchmark_data_smithi088_462306_object326:head# 2024-04-25T09:09:56.989 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:86a2ceef:::benchmark_data_smithi088_467817_object11:head# 2024-04-25T09:09:56.989 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8770f696:::benchmark_data_smithi088_462306_object253:head# 2024-04-25T09:09:56.990 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87b0211e:::benchmark_data_smithi088_462306_object8:head# 2024-04-25T09:09:56.990 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87cd1abd:::benchmark_data_smithi088_462306_object128:head# 2024-04-25T09:09:56.991 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87cf274d:::benchmark_data_smithi088_462306_object19:head# 2024-04-25T09:09:56.991 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87f67271:::benchmark_data_smithi088_462306_object189:head# 2024-04-25T09:09:56.992 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8802f69d:::benchmark_data_smithi088_467817_object26:head# 2024-04-25T09:09:56.992 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:88f3d2ff:::benchmark_data_smithi088_462306_object173:head# 2024-04-25T09:09:56.993 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8a4612ce:::benchmark_data_smithi088_462306_object233:head# 2024-04-25T09:09:56.993 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8a69a4db:::benchmark_data_smithi088_467817_object120:head# 2024-04-25T09:09:56.993 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8aa12c5c:::benchmark_data_smithi088_467817_object104:head# 2024-04-25T09:09:56.994 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8bb6f309:::benchmark_data_smithi088_462306_object132:head# 2024-04-25T09:09:56.995 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8ca22c61:::benchmark_data_smithi088_462306_object220:head# 2024-04-25T09:09:56.995 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8ce82677:::benchmark_data_smithi088_467817_object93:head# 2024-04-25T09:09:56.995 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8cfffb40:::benchmark_data_smithi088_467817_object16:head# 2024-04-25T09:09:56.996 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8d801fc0:::benchmark_data_smithi088_462306_object207:head# 2024-04-25T09:09:56.996 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8fbb559d:::benchmark_data_smithi088_462306_object257:head# 2024-04-25T09:09:56.997 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:91623a25:::benchmark_data_smithi088_462306_object138:head# 2024-04-25T09:09:56.997 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9196f898:::benchmark_data_smithi088_462306_object6:head# 2024-04-25T09:09:56.998 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:91e42837:::benchmark_data_smithi088_462306_object176:head# 2024-04-25T09:09:56.998 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:91e98fd5:::benchmark_data_smithi088_462306_object35:head# 2024-04-25T09:09:56.999 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:92220cfe:::benchmark_data_smithi088_462306_object22:head# 2024-04-25T09:09:56.999 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9265404e:::benchmark_data_smithi088_462306_object169:head# 2024-04-25T09:09:56.999 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:92ad4e27:::benchmark_data_smithi088_462306_object238:head# 2024-04-25T09:09:57.000 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:934c91ea:::benchmark_data_smithi088_462306_object246:head# 2024-04-25T09:09:57.000 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:937e9a49:::benchmark_data_smithi088_462306_object199:head# 2024-04-25T09:09:57.001 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:93b8c46b:::benchmark_data_smithi088_462306_object76:head# 2024-04-25T09:09:57.001 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9411c558:::benchmark_data_smithi088_462306_object24:head# 2024-04-25T09:09:57.002 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:94abe535:::benchmark_data_smithi088_462306_object60:head# 2024-04-25T09:09:57.002 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:96cf9e3a:::benchmark_data_smithi088_462306_object325:head# 2024-04-25T09:09:57.002 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:981a16b9:::benchmark_data_smithi088_467817_object53:head# 2024-04-25T09:09:57.003 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:98888887:::benchmark_data_smithi088_462306_object130:head# 2024-04-25T09:09:57.003 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:98cc271a:::benchmark_data_smithi088_462306_object174:head# 2024-04-25T09:09:57.004 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9935b1ed:::benchmark_data_smithi088_462306_object272:head# 2024-04-25T09:09:57.004 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9938cea0:::benchmark_data_smithi088_462306_object162:head# 2024-04-25T09:09:57.005 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:99c8e7d5:::benchmark_data_smithi088_467817_object36:head# 2024-04-25T09:09:57.005 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9acb5316:::benchmark_data_smithi088_462306_object222:head# 2024-04-25T09:09:57.006 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9b089756:::benchmark_data_smithi088_462306_object271:head# 2024-04-25T09:09:57.006 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9baeb73f:::benchmark_data_smithi088_462306_object117:head# 2024-04-25T09:09:57.007 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9bf57a33:::benchmark_data_smithi088_462306_object204:head# 2024-04-25T09:09:57.007 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9bf7ace7:::benchmark_data_smithi088_467817_object72:head# 2024-04-25T09:09:57.007 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9c2d402a:::benchmark_data_smithi088_467817_object66:head# 2024-04-25T09:09:57.008 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9c595d90:::benchmark_data_smithi088_462306_object16:head# 2024-04-25T09:09:57.008 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9caedf93:::benchmark_data_smithi088_462306_object109:head# 2024-04-25T09:09:57.009 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9cb158b2:::benchmark_data_smithi088_462306_object150:head# 2024-04-25T09:09:57.009 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9cb772db:::benchmark_data_smithi088_467817_object9:head# 2024-04-25T09:09:57.009 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9cc125fe:::benchmark_data_smithi088_467817_object1:head# 2024-04-25T09:09:57.010 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9cd16357:::benchmark_data_smithi088_467817_object5:head# 2024-04-25T09:09:57.010 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9f6e371a:::benchmark_data_smithi088_462306_object275:head# 2024-04-25T09:09:57.011 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a20735fd:::benchmark_data_smithi088_462306_object282:head# 2024-04-25T09:09:57.011 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a2c8e7c0:::benchmark_data_smithi088_462306_object116:head# 2024-04-25T09:09:57.012 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a3548fec:::benchmark_data_smithi088_462306_object178:head# 2024-04-25T09:09:57.012 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a3d97e4f:::benchmark_data_smithi088_462306_object37:head# 2024-04-25T09:09:57.013 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a40b440b:::benchmark_data_smithi088_462306_object197:head# 2024-04-25T09:09:57.013 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a4fe7d56:::benchmark_data_smithi088_462306_object92:head# 2024-04-25T09:09:57.013 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6c17891:::benchmark_data_smithi088_462306_object146:head# 2024-04-25T09:09:57.014 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6cdc671:::benchmark_data_smithi088_462306_object195:head# 2024-04-25T09:09:57.014 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6ebe5bf:::benchmark_data_smithi088_462306_object145:head# 2024-04-25T09:09:57.015 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6ef9d72:::benchmark_data_smithi088_467817_object61:head# 2024-04-25T09:09:57.015 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a7506a6e:::benchmark_data_smithi088_462306_object122:head# 2024-04-25T09:09:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a87a78a1:::benchmark_data_smithi088_462306_object5:head# 2024-04-25T09:09:57.016 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a87bb1e0:::benchmark_data_smithi088_462306_object263:head# 2024-04-25T09:09:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a8de3fc3:::benchmark_data_smithi088_462306_object134:head# 2024-04-25T09:09:57.026 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a92036b2:::benchmark_data_smithi088_462306_object225:head# 2024-04-25T09:09:57.027 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:aa84cf26:::benchmark_data_smithi088_462306_object11:head# 2024-04-25T09:09:57.028 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:aaac40ab:::benchmark_data_smithi088_467817_object125:head# 2024-04-25T09:09:57.030 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ab034bbd:::benchmark_data_smithi088_462306_object232:head# 2024-04-25T09:09:57.030 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ab565d7b:::benchmark_data_smithi088_462306_object302:head# 2024-04-25T09:09:57.031 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:abc81ff6:::benchmark_data_smithi088_462306_object69:head# 2024-04-25T09:09:57.032 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ac08c85b:::benchmark_data_smithi088_462306_object61:head# 2024-04-25T09:09:57.032 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:acecab62:::benchmark_data_smithi088_462306_object219:head# 2024-04-25T09:09:57.033 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ad2c69f2:::benchmark_data_smithi088_467817_object13:head# 2024-04-25T09:09:57.034 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ad48ca91:::benchmark_data_smithi088_462306_object209:head# 2024-04-25T09:09:57.034 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:af94d379:::benchmark_data_smithi088_462306_object298:head# 2024-04-25T09:09:57.035 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b0688a46:::benchmark_data_smithi088_467817_object51:head# 2024-04-25T09:09:57.035 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b06befdd:::benchmark_data_smithi088_462306_object156:head# 2024-04-25T09:09:57.036 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b0911968:::benchmark_data_smithi088_467817_object123:head# 2024-04-25T09:09:57.036 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b0c2cf35:::benchmark_data_smithi088_462306_object240:head# 2024-04-25T09:09:57.037 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b0e3af7f:::benchmark_data_smithi088_467817_object12:head# 2024-04-25T09:09:57.037 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b17d8fef:::benchmark_data_smithi088_462306_object13:head# 2024-04-25T09:09:57.038 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b20bd2ac:::benchmark_data_smithi088_462306_object26:head# 2024-04-25T09:09:57.038 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b2160c47:::benchmark_data_smithi088_462306_object159:head# 2024-04-25T09:09:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b2850783:::benchmark_data_smithi088_467817_object80:head# 2024-04-25T09:09:57.039 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b2f527bb:::benchmark_data_smithi088_462306_object21:head# 2024-04-25T09:09:57.040 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b32d563a:::benchmark_data_smithi088_467817_object34:head# 2024-04-25T09:09:57.040 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b35e628e:::benchmark_data_smithi088_467817_object91:head# 2024-04-25T09:09:57.041 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b38816db:::benchmark_data_smithi088_462306_object87:head# 2024-04-25T09:09:57.041 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b39555ce:::benchmark_data_smithi088_462306_object311:head# 2024-04-25T09:09:57.042 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b3d4574c:::benchmark_data_smithi088_462306_object182:head# 2024-04-25T09:09:57.047 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b50b3477:::benchmark_data_smithi088_462306_object55:head# 2024-04-25T09:09:57.047 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b5453a8b:::benchmark_data_smithi088_467817_object59:head# 2024-04-25T09:09:57.047 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b65ada81:::benchmark_data_smithi088_467817_object43:head# 2024-04-25T09:09:57.047 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b7cd0d37:::benchmark_data_smithi088_462306_object294:head# 2024-04-25T09:09:57.047 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b82e2947:::benchmark_data_smithi088_462306_object163:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b8db97a9:::benchmark_data_smithi088_462306_object165:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b97bb4a3:::benchmark_data_smithi088_462306_object319:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba2c6dac:::benchmark_data_smithi088_462306_object251:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba685cfd:::benchmark_data_smithi088_462306_object99:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba7c5b40:::benchmark_data_smithi088_462306_object202:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba8d78f2:::benchmark_data_smithi088_462306_object148:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:baf95fb7:::benchmark_data_smithi088_462306_object290:head# 2024-04-25T09:09:57.049 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bca58a03:::benchmark_data_smithi088_462306_object129:head# 2024-04-25T09:09:57.050 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bd7d893a:::benchmark_data_smithi088_467817_object30:head# 2024-04-25T09:09:57.050 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bdd50353:::benchmark_data_smithi088_462306_object98:head# 2024-04-25T09:09:57.051 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:be02a273:::benchmark_data_smithi088_462306_object291:head# 2024-04-25T09:09:57.051 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bf30e66b:::benchmark_data_smithi088_467817_object121:head# 2024-04-25T09:09:57.052 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bf8b18a3:::benchmark_data_smithi088_467817_object2:head# 2024-04-25T09:09:57.052 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bfa07270:::benchmark_data_smithi088_462306_object258:head# 2024-04-25T09:09:57.053 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bfc2ddbb:::benchmark_data_smithi088_462306_object155:head# 2024-04-25T09:09:57.054 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c18f632f:::benchmark_data_smithi088_462306_object43:head# 2024-04-25T09:09:57.054 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c1eedab1:::benchmark_data_smithi088_467817_object28:head# 2024-04-25T09:09:57.054 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c2f3461e:::benchmark_data_smithi088_462306_object91:head# 2024-04-25T09:09:57.055 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c3b33d4e:::benchmark_data_smithi088_467817_object23:head# 2024-04-25T09:09:57.056 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c3bd9ba8:::benchmark_data_smithi088_467817_object105:head# 2024-04-25T09:09:57.056 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c40c5699:::benchmark_data_smithi088_462306_object280:head# 2024-04-25T09:09:57.057 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c4517802:::benchmark_data_smithi088_462306_object3:head# 2024-04-25T09:09:57.058 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c4dec443:::benchmark_data_smithi088_462306_object277:head# 2024-04-25T09:09:57.058 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c588688f:::benchmark_data_smithi088_462306_object273:head# 2024-04-25T09:09:57.059 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c5ea4073:::benchmark_data_smithi088_467817_object81:head# 2024-04-25T09:09:57.059 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c621a2c5:::benchmark_data_smithi088_462306_object51:head# 2024-04-25T09:09:57.060 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c6b2539b:::benchmark_data_smithi088_462306_object254:head# 2024-04-25T09:09:57.061 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c8b8d985:::benchmark_data_smithi088_462306_object118:head# 2024-04-25T09:09:57.061 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c8e1a562:::benchmark_data_smithi088_462306_object245:head# 2024-04-25T09:09:57.062 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c9f5e653:::benchmark_data_smithi088_467817_object78:head# 2024-04-25T09:09:57.062 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ca40773e:::benchmark_data_smithi088_467817_object44:head# 2024-04-25T09:09:57.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cace5b80:::benchmark_data_smithi088_467817_object110:head# 2024-04-25T09:09:57.063 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cad804f9:::benchmark_data_smithi088_462306_object285:head# 2024-04-25T09:09:57.064 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cae4686f:::benchmark_data_smithi088_467817_object100:head# 2024-04-25T09:09:57.064 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cb0bb72b:::benchmark_data_smithi088_462306_object268:head# 2024-04-25T09:09:57.065 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cb9c1d39:::benchmark_data_smithi088_462306_object304:head# 2024-04-25T09:09:57.065 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cba404d8:::benchmark_data_smithi088_462306_object34:head# 2024-04-25T09:09:57.066 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cc8e65c8:::benchmark_data_smithi088_462306_object334:head# 2024-04-25T09:09:57.067 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cd3cc54c:::benchmark_data_smithi088_467817_object64:head# 2024-04-25T09:09:57.067 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cd75ee52:::benchmark_data_smithi088_462306_object172:head# 2024-04-25T09:09:57.067 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ce72978b:::benchmark_data_smithi088_462306_object226:head# 2024-04-25T09:09:57.068 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cebe6f8b:::benchmark_data_smithi088_462306_object242:head# 2024-04-25T09:09:57.068 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cf135ed8:::benchmark_data_smithi088_462306_object97:head# 2024-04-25T09:09:57.069 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cf1ea7f9:::benchmark_data_smithi088_462306_object337:head# 2024-04-25T09:09:57.070 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cfc2a95a:::benchmark_data_smithi088_462306_object119:head# 2024-04-25T09:09:57.070 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cffde5b4:::benchmark_data_smithi088_467817_object103:head# 2024-04-25T09:09:57.071 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d06b5209:::benchmark_data_smithi088_462306_object152:head# 2024-04-25T09:09:57.072 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d0efe327:::benchmark_data_smithi088_462306_object306:head# 2024-04-25T09:09:57.072 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d1723fce:::benchmark_data_smithi088_467817_object77:head# 2024-04-25T09:09:57.073 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d182b227:::benchmark_data_smithi088_462306_object100:head# 2024-04-25T09:09:57.074 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d1977b2c:::benchmark_data_smithi088_462306_object166:head# 2024-04-25T09:09:57.074 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d1e00a1c:::benchmark_data_smithi088_462306_object48:head# 2024-04-25T09:09:57.075 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d297305a:::benchmark_data_smithi088_462306_object301:head# 2024-04-25T09:09:57.075 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d2cbd392:::benchmark_data_smithi088_462306_object136:head# 2024-04-25T09:09:57.076 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d2dc1847:::benchmark_data_smithi088_462306_object36:head# 2024-04-25T09:09:57.076 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d2f0530a:::benchmark_data_smithi088_462306_object331:head# 2024-04-25T09:09:57.077 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d41367b1:::benchmark_data_smithi088_467817_object86:head# 2024-04-25T09:09:57.078 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d41d1c70:::benchmark_data_smithi088_462306_object183:head# 2024-04-25T09:09:57.078 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d42ba1fa:::benchmark_data_smithi088_462306_object38:head# 2024-04-25T09:09:57.079 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d42ca9ea:::benchmark_data_smithi088_462306_object259:head# 2024-04-25T09:09:57.079 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d4ccf510:::benchmark_data_smithi088_462306_object206:head# 2024-04-25T09:09:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d5ddf04f:::benchmark_data_smithi088_467817_object15:head# 2024-04-25T09:09:57.080 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d658cee6:::benchmark_data_smithi088_462306_object279:head# 2024-04-25T09:09:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d6791640:::benchmark_data_smithi088_462306_object9:head# 2024-04-25T09:09:57.081 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d69229b2:::benchmark_data_smithi088_462306_object312:head# 2024-04-25T09:09:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d746333f:::benchmark_data_smithi088_462306_object1:head# 2024-04-25T09:09:57.082 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d8820786:::benchmark_data_smithi088_462306_object336:head# 2024-04-25T09:09:57.083 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d8e6af19:::benchmark_data_smithi088_467817_object45:head# 2024-04-25T09:09:57.084 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d8e83737:::benchmark_data_smithi088_462306_object203:head# 2024-04-25T09:09:57.085 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d92e92c0:::benchmark_data_smithi088_462306_object153:head# 2024-04-25T09:09:57.085 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d9ed16a2:::benchmark_data_smithi088_462306_object244:head# 2024-04-25T09:09:57.086 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d9f9d37f:::benchmark_data_smithi088_462306_object322:head# 2024-04-25T09:09:57.086 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:da421750:::benchmark_data_smithi088_462306_object283:head# 2024-04-25T09:09:57.087 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dad3e1cc:::benchmark_data_smithi088_462306_object293:head# 2024-04-25T09:09:57.087 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dbf914c5:::benchmark_data_smithi088_462306_object78:head# 2024-04-25T09:09:57.088 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dbfcbc35:::benchmark_data_smithi088_462306_object41:head# 2024-04-25T09:09:57.088 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dc672535:::benchmark_data_smithi088_462306_object82:head# 2024-04-25T09:09:57.088 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dc835119:::benchmark_data_smithi088_467817_object74:head# 2024-04-25T09:09:57.089 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dd09b3db:::benchmark_data_smithi088_462306_object12:head# 2024-04-25T09:09:57.090 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:de8da2f7:::benchmark_data_smithi088_462306_object267:head# 2024-04-25T09:09:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:df136223:::benchmark_data_smithi088_462306_object81:head# 2024-04-25T09:09:57.091 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:df552722:::benchmark_data_smithi088_462306_object54:head# 2024-04-25T09:09:57.092 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dfd38933:::benchmark_data_smithi088_467817_object70:head# 2024-04-25T09:09:57.092 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e074f344:::benchmark_data_smithi088_462306_object231:head# 2024-04-25T09:09:57.093 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e0d77fb7:::benchmark_data_smithi088_462306_object229:head# 2024-04-25T09:09:57.093 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e0eb6167:::benchmark_data_smithi088_462306_object101:head# 2024-04-25T09:09:57.094 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e0fbd74b:::benchmark_data_smithi088_462306_object333:head# 2024-04-25T09:09:57.094 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e1956e90:::benchmark_data_smithi088_462306_object243:head# 2024-04-25T09:09:57.095 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e22f836d:::benchmark_data_smithi088_462306_object180:head# 2024-04-25T09:09:57.095 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e25ba6c9:::benchmark_data_smithi088_462306_object15:head# 2024-04-25T09:09:57.095 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e326ce95:::benchmark_data_smithi088_467817_object39:head# 2024-04-25T09:09:57.097 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e344d3d7:::benchmark_data_smithi088_467817_object69:head# 2024-04-25T09:09:57.097 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e3fbb848:::benchmark_data_smithi088_462306_object30:head# 2024-04-25T09:09:57.097 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e490c252:::benchmark_data_smithi088_462306_object281:head# 2024-04-25T09:09:57.098 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e4fb9482:::benchmark_data_smithi088_462306_object260:head# 2024-04-25T09:09:57.099 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e5649a8a:::benchmark_data_smithi088_467817_object38:head# 2024-04-25T09:09:57.099 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e5dae559:::benchmark_data_smithi088_462306_object170:head# 2024-04-25T09:09:57.100 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e665af06:::benchmark_data_smithi088_462306_object49:head# 2024-04-25T09:09:57.101 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e6cff169:::benchmark_data_smithi088_467817_object92:head# 2024-04-25T09:09:57.101 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e7b1daf4:::benchmark_data_smithi088_462306_object47:head# 2024-04-25T09:09:57.102 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e8c06845:::benchmark_data_smithi088_467817_object42:head# 2024-04-25T09:09:57.103 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e9823a3c:::benchmark_data_smithi088_467817_object56:head# 2024-04-25T09:09:57.103 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e9919842:::benchmark_data_smithi088_462306_object289:head# 2024-04-25T09:09:57.104 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ebc23c5b:::benchmark_data_smithi088_462306_object316:head# 2024-04-25T09:09:57.104 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ebddce23:::benchmark_data_smithi088_462306_object286:head# 2024-04-25T09:09:57.105 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:eccef000:::benchmark_data_smithi088_467817_object122:head# 2024-04-25T09:09:57.106 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ed2ff011:::benchmark_data_smithi088_467817_object111:head# 2024-04-25T09:09:57.106 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ef8b7d88:::benchmark_data_smithi088_467817_object10:head# 2024-04-25T09:09:57.107 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f01c7538:::benchmark_data_smithi088_462306_object255:head# 2024-04-25T09:09:57.107 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f03d3cff:::benchmark_data_smithi088_462306_object184:head# 2024-04-25T09:09:57.108 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f0516783:::benchmark_data_smithi088_467817_object35:head# 2024-04-25T09:09:57.108 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f07d123f:::benchmark_data_smithi088_467817_object49:head# 2024-04-25T09:09:57.109 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f0b18d8c:::benchmark_data_smithi088_462306_object7:head# 2024-04-25T09:09:57.109 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f2319ef1:::benchmark_data_smithi088_462306_object46:head# 2024-04-25T09:09:57.110 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f27dc48a:::benchmark_data_smithi088_462306_object175:head# 2024-04-25T09:09:57.110 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f2e14e58:::benchmark_data_smithi088_467817_object102:head# 2024-04-25T09:09:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f3cf7b95:::benchmark_data_smithi088_462306_object112:head# 2024-04-25T09:09:57.112 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f3f2f56d:::benchmark_data_smithi088_467817_object63:head# 2024-04-25T09:09:57.112 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f40a2c8a:::benchmark_data_smithi088_467817_object65:head# 2024-04-25T09:09:57.113 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f473732d:::benchmark_data_smithi088_462306_object265:head# 2024-04-25T09:09:57.113 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f4b5afc3:::benchmark_data_smithi088_467817_object24:head# 2024-04-25T09:09:57.114 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f50d52f3:::benchmark_data_smithi088_462306_object227:head# 2024-04-25T09:09:57.114 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f69dfeef:::benchmark_data_smithi088_462306_object186:head# 2024-04-25T09:09:57.115 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f77a4b03:::benchmark_data_smithi088_467817_object113:head# 2024-04-25T09:09:57.115 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f7adaaf0:::benchmark_data_smithi088_462306_object214:head# 2024-04-25T09:09:57.116 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f7c73560:::benchmark_data_smithi088_462306_object332:head# 2024-04-25T09:09:57.116 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f8160d49:::benchmark_data_smithi088_462306_object314:head# 2024-04-25T09:09:57.117 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f90ccab7:::benchmark_data_smithi088_462306_object142:head# 2024-04-25T09:09:57.118 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f91a495a:::benchmark_data_smithi088_462306_object72:head# 2024-04-25T09:09:57.118 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fabfa17e:::benchmark_data_smithi088_462306_object140:head# 2024-04-25T09:09:57.119 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fb87786e:::benchmark_data_smithi088_462306_object160:head# 2024-04-25T09:09:57.119 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fbe0687a:::benchmark_data_smithi088_462306_object230:head# 2024-04-25T09:09:57.120 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fc115286:::benchmark_data_smithi088_462306_object39:head# 2024-04-25T09:09:57.120 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fd7dad8e:::benchmark_data_smithi088_467817_object95:head# 2024-04-25T09:09:57.121 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fe11890a:::benchmark_data_smithi088_467817_object114:head# 2024-04-25T09:09:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fe55e075:::benchmark_data_smithi088_462306_object335:head# 2024-04-25T09:09:57.122 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fefad482:::benchmark_data_smithi088_462306_object79:head# 2024-04-25T09:09:57.123 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ff2bd568:::benchmark_data_smithi088_462306_object90:head# 2024-04-25T09:09:57.124 INFO:tasks.workunit.client.0.smithi088.stderr:Export successful 2024-04-25T09:09:57.124 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T09:09:57.124 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T09:09:57.124 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:09:57.550 INFO:tasks.workunit.client.0.smithi088.stderr:Remove successful 2024-04-25T09:09:57.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:143: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:09:58.651 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e227 pg_nums {1={18=1}} deleted_pools ) 2024-04-25T09:09:58.651 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.1 2024-04-25T09:09:58.651 INFO:tasks.workunit.client.0.smithi088.stderr:PG export's map 8 is older than OSD's oldest_map 61 2024-04-25T09:09:58.652 INFO:tasks.workunit.client.0.smithi088.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:09:59.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:144: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:09:59.669 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e227 pg_nums {1={18=1}} deleted_pools ) 2024-04-25T09:09:59.669 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.0 2024-04-25T09:09:59.670 INFO:tasks.workunit.client.0.smithi088.stderr:PG export's map 8 is older than OSD's oldest_map 61 2024-04-25T09:09:59.670 INFO:tasks.workunit.client.0.smithi088.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:10:00.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:147: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-04-25T09:10:00.688 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e227 pg_nums {1={18=1}} deleted_pools ) 2024-04-25T09:10:00.688 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.1 2024-04-25T09:10:00.688 INFO:tasks.workunit.client.0.smithi088.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:10:00.689 INFO:tasks.workunit.client.0.smithi088.stderr:PG export's map 8 is older than OSD's oldest_map 61 2024-04-25T09:10:00.689 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81099aed:::benchmark_data_smithi088_462306_object181:head# 2024-04-25T09:10:00.689 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.709 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81ae27f9:::benchmark_data_smithi088_462306_object252:head# 2024-04-25T09:10:00.709 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.717 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81b12e8e:::benchmark_data_smithi088_462306_object161:head# 2024-04-25T09:10:00.717 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.726 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81e3c822:::benchmark_data_smithi088_462306_object64:head# 2024-04-25T09:10:00.726 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.734 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:82743540:::benchmark_data_smithi088_462306_object276:head# 2024-04-25T09:10:00.734 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.742 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:842cb2a9:::benchmark_data_smithi088_462306_object278:head# 2024-04-25T09:10:00.742 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.751 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:84c54114:::benchmark_data_smithi088_462306_object66:head# 2024-04-25T09:10:00.751 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.759 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:85078920:::benchmark_data_smithi088_462306_object62:head# 2024-04-25T09:10:00.759 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.767 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:85f7c2db:::benchmark_data_smithi088_462306_object326:head# 2024-04-25T09:10:00.767 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.776 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8770f696:::benchmark_data_smithi088_462306_object253:head# 2024-04-25T09:10:00.776 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.784 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87b0211e:::benchmark_data_smithi088_462306_object8:head# 2024-04-25T09:10:00.784 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.792 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87cd1abd:::benchmark_data_smithi088_462306_object128:head# 2024-04-25T09:10:00.792 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.801 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87cf274d:::benchmark_data_smithi088_462306_object19:head# 2024-04-25T09:10:00.801 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.809 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87f67271:::benchmark_data_smithi088_462306_object189:head# 2024-04-25T09:10:00.809 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.817 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:88f3d2ff:::benchmark_data_smithi088_462306_object173:head# 2024-04-25T09:10:00.817 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.826 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8a4612ce:::benchmark_data_smithi088_462306_object233:head# 2024-04-25T09:10:00.826 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.834 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8bb6f309:::benchmark_data_smithi088_462306_object132:head# 2024-04-25T09:10:00.834 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.842 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8ca22c61:::benchmark_data_smithi088_462306_object220:head# 2024-04-25T09:10:00.842 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.851 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8d801fc0:::benchmark_data_smithi088_462306_object207:head# 2024-04-25T09:10:00.851 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.859 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8fbb559d:::benchmark_data_smithi088_462306_object257:head# 2024-04-25T09:10:00.859 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.867 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91623a25:::benchmark_data_smithi088_462306_object138:head# 2024-04-25T09:10:00.868 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.876 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9196f898:::benchmark_data_smithi088_462306_object6:head# 2024-04-25T09:10:00.876 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.884 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91e42837:::benchmark_data_smithi088_462306_object176:head# 2024-04-25T09:10:00.884 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.892 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91e98fd5:::benchmark_data_smithi088_462306_object35:head# 2024-04-25T09:10:00.893 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.901 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92220cfe:::benchmark_data_smithi088_462306_object22:head# 2024-04-25T09:10:00.901 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.909 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9265404e:::benchmark_data_smithi088_462306_object169:head# 2024-04-25T09:10:00.909 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.918 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92ad4e27:::benchmark_data_smithi088_462306_object238:head# 2024-04-25T09:10:00.918 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.926 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:934c91ea:::benchmark_data_smithi088_462306_object246:head# 2024-04-25T09:10:00.926 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.934 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:937e9a49:::benchmark_data_smithi088_462306_object199:head# 2024-04-25T09:10:00.934 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.943 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:93b8c46b:::benchmark_data_smithi088_462306_object76:head# 2024-04-25T09:10:00.943 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.951 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9411c558:::benchmark_data_smithi088_462306_object24:head# 2024-04-25T09:10:00.951 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.959 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:94abe535:::benchmark_data_smithi088_462306_object60:head# 2024-04-25T09:10:00.959 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.968 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:96cf9e3a:::benchmark_data_smithi088_462306_object325:head# 2024-04-25T09:10:00.968 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.976 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:98888887:::benchmark_data_smithi088_462306_object130:head# 2024-04-25T09:10:00.976 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.985 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:98cc271a:::benchmark_data_smithi088_462306_object174:head# 2024-04-25T09:10:00.985 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:00.993 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9935b1ed:::benchmark_data_smithi088_462306_object272:head# 2024-04-25T09:10:00.993 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.001 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9938cea0:::benchmark_data_smithi088_462306_object162:head# 2024-04-25T09:10:01.001 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.009 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9acb5316:::benchmark_data_smithi088_462306_object222:head# 2024-04-25T09:10:01.010 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.018 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9b089756:::benchmark_data_smithi088_462306_object271:head# 2024-04-25T09:10:01.018 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.026 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9baeb73f:::benchmark_data_smithi088_462306_object117:head# 2024-04-25T09:10:01.026 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.034 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9bf57a33:::benchmark_data_smithi088_462306_object204:head# 2024-04-25T09:10:01.035 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.043 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9c595d90:::benchmark_data_smithi088_462306_object16:head# 2024-04-25T09:10:01.043 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.051 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9caedf93:::benchmark_data_smithi088_462306_object109:head# 2024-04-25T09:10:01.051 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.060 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9cb158b2:::benchmark_data_smithi088_462306_object150:head# 2024-04-25T09:10:01.060 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.068 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9f6e371a:::benchmark_data_smithi088_462306_object275:head# 2024-04-25T09:10:01.068 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.076 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a20735fd:::benchmark_data_smithi088_462306_object282:head# 2024-04-25T09:10:01.076 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.085 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a2c8e7c0:::benchmark_data_smithi088_462306_object116:head# 2024-04-25T09:10:01.085 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.093 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a3548fec:::benchmark_data_smithi088_462306_object178:head# 2024-04-25T09:10:01.093 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.101 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a3d97e4f:::benchmark_data_smithi088_462306_object37:head# 2024-04-25T09:10:01.101 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.110 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a40b440b:::benchmark_data_smithi088_462306_object197:head# 2024-04-25T09:10:01.110 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.118 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a4fe7d56:::benchmark_data_smithi088_462306_object92:head# 2024-04-25T09:10:01.118 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.126 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6c17891:::benchmark_data_smithi088_462306_object146:head# 2024-04-25T09:10:01.126 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.135 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6cdc671:::benchmark_data_smithi088_462306_object195:head# 2024-04-25T09:10:01.135 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.143 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6ebe5bf:::benchmark_data_smithi088_462306_object145:head# 2024-04-25T09:10:01.143 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.151 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a7506a6e:::benchmark_data_smithi088_462306_object122:head# 2024-04-25T09:10:01.152 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.160 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a87a78a1:::benchmark_data_smithi088_462306_object5:head# 2024-04-25T09:10:01.160 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.168 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a87bb1e0:::benchmark_data_smithi088_462306_object263:head# 2024-04-25T09:10:01.168 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.176 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a8de3fc3:::benchmark_data_smithi088_462306_object134:head# 2024-04-25T09:10:01.177 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.185 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a92036b2:::benchmark_data_smithi088_462306_object225:head# 2024-04-25T09:10:01.185 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.193 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:aa84cf26:::benchmark_data_smithi088_462306_object11:head# 2024-04-25T09:10:01.193 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.202 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab034bbd:::benchmark_data_smithi088_462306_object232:head# 2024-04-25T09:10:01.202 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.210 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab565d7b:::benchmark_data_smithi088_462306_object302:head# 2024-04-25T09:10:01.210 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.218 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:abc81ff6:::benchmark_data_smithi088_462306_object69:head# 2024-04-25T09:10:01.218 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.227 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ac08c85b:::benchmark_data_smithi088_462306_object61:head# 2024-04-25T09:10:01.227 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.235 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:acecab62:::benchmark_data_smithi088_462306_object219:head# 2024-04-25T09:10:01.235 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.260 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ad48ca91:::benchmark_data_smithi088_462306_object209:head# 2024-04-25T09:10:01.260 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.268 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:af94d379:::benchmark_data_smithi088_462306_object298:head# 2024-04-25T09:10:01.269 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.277 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b06befdd:::benchmark_data_smithi088_462306_object156:head# 2024-04-25T09:10:01.277 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.285 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b0c2cf35:::benchmark_data_smithi088_462306_object240:head# 2024-04-25T09:10:01.285 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.294 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b17d8fef:::benchmark_data_smithi088_462306_object13:head# 2024-04-25T09:10:01.294 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.302 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b20bd2ac:::benchmark_data_smithi088_462306_object26:head# 2024-04-25T09:10:01.302 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.310 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b2160c47:::benchmark_data_smithi088_462306_object159:head# 2024-04-25T09:10:01.310 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.318 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b2f527bb:::benchmark_data_smithi088_462306_object21:head# 2024-04-25T09:10:01.319 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.327 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b38816db:::benchmark_data_smithi088_462306_object87:head# 2024-04-25T09:10:01.327 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.337 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b39555ce:::benchmark_data_smithi088_462306_object311:head# 2024-04-25T09:10:01.338 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.343 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b3d4574c:::benchmark_data_smithi088_462306_object182:head# 2024-04-25T09:10:01.344 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.352 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b50b3477:::benchmark_data_smithi088_462306_object55:head# 2024-04-25T09:10:01.352 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.360 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b7cd0d37:::benchmark_data_smithi088_462306_object294:head# 2024-04-25T09:10:01.360 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.369 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b82e2947:::benchmark_data_smithi088_462306_object163:head# 2024-04-25T09:10:01.369 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.377 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:10:01.377 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.385 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b8db97a9:::benchmark_data_smithi088_462306_object165:head# 2024-04-25T09:10:01.385 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.394 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b97bb4a3:::benchmark_data_smithi088_462306_object319:head# 2024-04-25T09:10:01.394 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.402 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba2c6dac:::benchmark_data_smithi088_462306_object251:head# 2024-04-25T09:10:01.402 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.410 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba685cfd:::benchmark_data_smithi088_462306_object99:head# 2024-04-25T09:10:01.410 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.419 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba7c5b40:::benchmark_data_smithi088_462306_object202:head# 2024-04-25T09:10:01.419 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.435 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba8d78f2:::benchmark_data_smithi088_462306_object148:head# 2024-04-25T09:10:01.436 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.452 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:baf95fb7:::benchmark_data_smithi088_462306_object290:head# 2024-04-25T09:10:01.452 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.460 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bca58a03:::benchmark_data_smithi088_462306_object129:head# 2024-04-25T09:10:01.460 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.469 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bdd50353:::benchmark_data_smithi088_462306_object98:head# 2024-04-25T09:10:01.469 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.477 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:be02a273:::benchmark_data_smithi088_462306_object291:head# 2024-04-25T09:10:01.477 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.485 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bfa07270:::benchmark_data_smithi088_462306_object258:head# 2024-04-25T09:10:01.486 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.494 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bfc2ddbb:::benchmark_data_smithi088_462306_object155:head# 2024-04-25T09:10:01.494 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.502 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c18f632f:::benchmark_data_smithi088_462306_object43:head# 2024-04-25T09:10:01.502 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.510 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c2f3461e:::benchmark_data_smithi088_462306_object91:head# 2024-04-25T09:10:01.511 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.519 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c40c5699:::benchmark_data_smithi088_462306_object280:head# 2024-04-25T09:10:01.519 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.527 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c4517802:::benchmark_data_smithi088_462306_object3:head# 2024-04-25T09:10:01.527 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.536 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c4dec443:::benchmark_data_smithi088_462306_object277:head# 2024-04-25T09:10:01.536 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.578 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c588688f:::benchmark_data_smithi088_462306_object273:head# 2024-04-25T09:10:01.578 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.586 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c621a2c5:::benchmark_data_smithi088_462306_object51:head# 2024-04-25T09:10:01.586 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.594 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c6b2539b:::benchmark_data_smithi088_462306_object254:head# 2024-04-25T09:10:01.594 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.602 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c8b8d985:::benchmark_data_smithi088_462306_object118:head# 2024-04-25T09:10:01.602 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.611 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c8e1a562:::benchmark_data_smithi088_462306_object245:head# 2024-04-25T09:10:01.611 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.619 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cad804f9:::benchmark_data_smithi088_462306_object285:head# 2024-04-25T09:10:01.619 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.628 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cb0bb72b:::benchmark_data_smithi088_462306_object268:head# 2024-04-25T09:10:01.628 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.636 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cb9c1d39:::benchmark_data_smithi088_462306_object304:head# 2024-04-25T09:10:01.636 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.644 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cba404d8:::benchmark_data_smithi088_462306_object34:head# 2024-04-25T09:10:01.644 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.653 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cc8e65c8:::benchmark_data_smithi088_462306_object334:head# 2024-04-25T09:10:01.653 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.661 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cd75ee52:::benchmark_data_smithi088_462306_object172:head# 2024-04-25T09:10:01.661 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.669 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ce72978b:::benchmark_data_smithi088_462306_object226:head# 2024-04-25T09:10:01.669 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.678 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cebe6f8b:::benchmark_data_smithi088_462306_object242:head# 2024-04-25T09:10:01.678 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.686 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cf135ed8:::benchmark_data_smithi088_462306_object97:head# 2024-04-25T09:10:01.686 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.694 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cf1ea7f9:::benchmark_data_smithi088_462306_object337:head# 2024-04-25T09:10:01.694 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.703 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cfc2a95a:::benchmark_data_smithi088_462306_object119:head# 2024-04-25T09:10:01.703 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.711 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d06b5209:::benchmark_data_smithi088_462306_object152:head# 2024-04-25T09:10:01.711 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.719 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d0efe327:::benchmark_data_smithi088_462306_object306:head# 2024-04-25T09:10:01.719 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.728 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d182b227:::benchmark_data_smithi088_462306_object100:head# 2024-04-25T09:10:01.728 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.736 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d1977b2c:::benchmark_data_smithi088_462306_object166:head# 2024-04-25T09:10:01.736 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.746 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d1e00a1c:::benchmark_data_smithi088_462306_object48:head# 2024-04-25T09:10:01.746 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.753 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d297305a:::benchmark_data_smithi088_462306_object301:head# 2024-04-25T09:10:01.753 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.761 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2cbd392:::benchmark_data_smithi088_462306_object136:head# 2024-04-25T09:10:01.761 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.769 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2dc1847:::benchmark_data_smithi088_462306_object36:head# 2024-04-25T09:10:01.770 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.778 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2f0530a:::benchmark_data_smithi088_462306_object331:head# 2024-04-25T09:10:01.778 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.786 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d41d1c70:::benchmark_data_smithi088_462306_object183:head# 2024-04-25T09:10:01.786 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.795 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d42ba1fa:::benchmark_data_smithi088_462306_object38:head# 2024-04-25T09:10:01.795 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.803 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d42ca9ea:::benchmark_data_smithi088_462306_object259:head# 2024-04-25T09:10:01.803 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.811 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d4ccf510:::benchmark_data_smithi088_462306_object206:head# 2024-04-25T09:10:01.811 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.820 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d658cee6:::benchmark_data_smithi088_462306_object279:head# 2024-04-25T09:10:01.820 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.828 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d6791640:::benchmark_data_smithi088_462306_object9:head# 2024-04-25T09:10:01.828 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.836 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d69229b2:::benchmark_data_smithi088_462306_object312:head# 2024-04-25T09:10:01.836 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.878 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d746333f:::benchmark_data_smithi088_462306_object1:head# 2024-04-25T09:10:01.878 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.886 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d8820786:::benchmark_data_smithi088_462306_object336:head# 2024-04-25T09:10:01.886 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.895 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d8e83737:::benchmark_data_smithi088_462306_object203:head# 2024-04-25T09:10:01.895 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.903 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d92e92c0:::benchmark_data_smithi088_462306_object153:head# 2024-04-25T09:10:01.903 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:01.949 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d9ed16a2:::benchmark_data_smithi088_462306_object244:head# 2024-04-25T09:10:01.949 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.000 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d9f9d37f:::benchmark_data_smithi088_462306_object322:head# 2024-04-25T09:10:02.000 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.011 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:da421750:::benchmark_data_smithi088_462306_object283:head# 2024-04-25T09:10:02.011 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.020 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dad3e1cc:::benchmark_data_smithi088_462306_object293:head# 2024-04-25T09:10:02.020 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.028 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dbf914c5:::benchmark_data_smithi088_462306_object78:head# 2024-04-25T09:10:02.028 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.036 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dbfcbc35:::benchmark_data_smithi088_462306_object41:head# 2024-04-25T09:10:02.037 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.045 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dc672535:::benchmark_data_smithi088_462306_object82:head# 2024-04-25T09:10:02.045 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.053 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd09b3db:::benchmark_data_smithi088_462306_object12:head# 2024-04-25T09:10:02.053 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.061 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:de8da2f7:::benchmark_data_smithi088_462306_object267:head# 2024-04-25T09:10:02.062 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.070 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:df136223:::benchmark_data_smithi088_462306_object81:head# 2024-04-25T09:10:02.070 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.078 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:df552722:::benchmark_data_smithi088_462306_object54:head# 2024-04-25T09:10:02.078 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.087 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e074f344:::benchmark_data_smithi088_462306_object231:head# 2024-04-25T09:10:02.087 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.095 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0d77fb7:::benchmark_data_smithi088_462306_object229:head# 2024-04-25T09:10:02.095 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.125 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0eb6167:::benchmark_data_smithi088_462306_object101:head# 2024-04-25T09:10:02.125 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.175 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0fbd74b:::benchmark_data_smithi088_462306_object333:head# 2024-04-25T09:10:02.175 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.217 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e1956e90:::benchmark_data_smithi088_462306_object243:head# 2024-04-25T09:10:02.217 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.267 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e22f836d:::benchmark_data_smithi088_462306_object180:head# 2024-04-25T09:10:02.267 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.317 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e25ba6c9:::benchmark_data_smithi088_462306_object15:head# 2024-04-25T09:10:02.317 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.328 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e3fbb848:::benchmark_data_smithi088_462306_object30:head# 2024-04-25T09:10:02.329 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.367 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e490c252:::benchmark_data_smithi088_462306_object281:head# 2024-04-25T09:10:02.368 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.427 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e4fb9482:::benchmark_data_smithi088_462306_object260:head# 2024-04-25T09:10:02.427 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.501 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e5dae559:::benchmark_data_smithi088_462306_object170:head# 2024-04-25T09:10:02.501 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.543 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e665af06:::benchmark_data_smithi088_462306_object49:head# 2024-04-25T09:10:02.543 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.587 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e7b1daf4:::benchmark_data_smithi088_462306_object47:head# 2024-04-25T09:10:02.587 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.628 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e9919842:::benchmark_data_smithi088_462306_object289:head# 2024-04-25T09:10:02.628 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ebc23c5b:::benchmark_data_smithi088_462306_object316:head# 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.645 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ebddce23:::benchmark_data_smithi088_462306_object286:head# 2024-04-25T09:10:02.645 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.653 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f01c7538:::benchmark_data_smithi088_462306_object255:head# 2024-04-25T09:10:02.653 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.662 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f03d3cff:::benchmark_data_smithi088_462306_object184:head# 2024-04-25T09:10:02.662 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.678 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f0b18d8c:::benchmark_data_smithi088_462306_object7:head# 2024-04-25T09:10:02.679 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.687 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f2319ef1:::benchmark_data_smithi088_462306_object46:head# 2024-04-25T09:10:02.687 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.695 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f27dc48a:::benchmark_data_smithi088_462306_object175:head# 2024-04-25T09:10:02.695 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.704 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f3cf7b95:::benchmark_data_smithi088_462306_object112:head# 2024-04-25T09:10:02.704 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.712 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f473732d:::benchmark_data_smithi088_462306_object265:head# 2024-04-25T09:10:02.712 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.729 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f50d52f3:::benchmark_data_smithi088_462306_object227:head# 2024-04-25T09:10:02.729 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.745 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f69dfeef:::benchmark_data_smithi088_462306_object186:head# 2024-04-25T09:10:02.745 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.762 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f7adaaf0:::benchmark_data_smithi088_462306_object214:head# 2024-04-25T09:10:02.762 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.779 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f7c73560:::benchmark_data_smithi088_462306_object332:head# 2024-04-25T09:10:02.779 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.787 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f8160d49:::benchmark_data_smithi088_462306_object314:head# 2024-04-25T09:10:02.787 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.804 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f90ccab7:::benchmark_data_smithi088_462306_object142:head# 2024-04-25T09:10:02.804 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.812 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f91a495a:::benchmark_data_smithi088_462306_object72:head# 2024-04-25T09:10:02.812 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.829 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fabfa17e:::benchmark_data_smithi088_462306_object140:head# 2024-04-25T09:10:02.829 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.845 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fb87786e:::benchmark_data_smithi088_462306_object160:head# 2024-04-25T09:10:02.846 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.854 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fbe0687a:::benchmark_data_smithi088_462306_object230:head# 2024-04-25T09:10:02.854 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.870 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fc115286:::benchmark_data_smithi088_462306_object39:head# 2024-04-25T09:10:02.870 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.887 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fe55e075:::benchmark_data_smithi088_462306_object335:head# 2024-04-25T09:10:02.887 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.904 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fefad482:::benchmark_data_smithi088_462306_object79:head# 2024-04-25T09:10:02.904 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.921 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ff2bd568:::benchmark_data_smithi088_462306_object90:head# 2024-04-25T09:10:02.921 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:02.937 INFO:tasks.workunit.client.0.smithi088.stdout:write_pg epoch 8 info 1.1( v 8'181 (0'0,8'181] local-lis/les=6/7 n=181 ec=6/6 lis/c=6/6 les/c/f=7/7/0 sis=6) 2024-04-25T09:10:02.954 INFO:tasks.workunit.client.0.smithi088.stdout:Import successful 2024-04-25T09:10:03.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:148: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:10:04.271 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e227 pg_nums {1={18=1}} deleted_pools ) 2024-04-25T09:10:04.271 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.0 2024-04-25T09:10:04.272 INFO:tasks.workunit.client.0.smithi088.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:10:04.272 INFO:tasks.workunit.client.0.smithi088.stderr:PG export's map 8 is older than OSD's oldest_map 61 2024-04-25T09:10:04.272 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0054a4c5:::benchmark_data_smithi088_462306_object56:head# 2024-04-25T09:10:04.272 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.313 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0091abb9:::benchmark_data_smithi088_462306_object216:head# 2024-04-25T09:10:04.313 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.337 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:01a57ad0:::benchmark_data_smithi088_462306_object317:head# 2024-04-25T09:10:04.338 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.355 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:01e683c2:::benchmark_data_smithi088_462306_object106:head# 2024-04-25T09:10:04.356 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.385 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:021a8ccb:::benchmark_data_smithi088_462306_object108:head# 2024-04-25T09:10:04.385 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.438 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:022d1a7b:::benchmark_data_smithi088_462306_object223:head# 2024-04-25T09:10:04.438 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.511 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0496a541:::benchmark_data_smithi088_462306_object115:head# 2024-04-25T09:10:04.511 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.554 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:052189ea:::benchmark_data_smithi088_462306_object198:head# 2024-04-25T09:10:04.554 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.571 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:052d5fd4:::benchmark_data_smithi088_462306_object14:head# 2024-04-25T09:10:04.571 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.596 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:06462b6c:::benchmark_data_smithi088_462306_object237:head# 2024-04-25T09:10:04.596 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.614 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0665cd86:::benchmark_data_smithi088_462306_object266:head# 2024-04-25T09:10:04.614 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.638 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:08ef67f6:::benchmark_data_smithi088_462306_object167:head# 2024-04-25T09:10:04.639 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.663 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:09033334:::benchmark_data_smithi088_462306_object57:head# 2024-04-25T09:10:04.663 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.688 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a07ee66:::benchmark_data_smithi088_462306_object139:head# 2024-04-25T09:10:04.688 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.713 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a9dbfd3:::benchmark_data_smithi088_462306_object147:head# 2024-04-25T09:10:04.713 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.738 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b31df74:::benchmark_data_smithi088_462306_object103:head# 2024-04-25T09:10:04.738 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.755 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b35794e:::benchmark_data_smithi088_462306_object309:head# 2024-04-25T09:10:04.755 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.780 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b54d91a:::benchmark_data_smithi088_462306_object143:head# 2024-04-25T09:10:04.780 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.805 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b783152:::benchmark_data_smithi088_462306_object53:head# 2024-04-25T09:10:04.805 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.822 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0be5222f:::benchmark_data_smithi088_462306_object295:head# 2024-04-25T09:10:04.822 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.838 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0c2e7645:::benchmark_data_smithi088_462306_object65:head# 2024-04-25T09:10:04.838 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.847 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0d3bf401:::benchmark_data_smithi088_462306_object84:head# 2024-04-25T09:10:04.847 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.863 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0da5c762:::benchmark_data_smithi088_462306_object137:head# 2024-04-25T09:10:04.863 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.888 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0efc8997:::benchmark_data_smithi088_462306_object228:head# 2024-04-25T09:10:04.889 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.914 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0f3cc8d1:::benchmark_data_smithi088_462306_object105:head# 2024-04-25T09:10:04.914 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.922 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0f6ddd44:::benchmark_data_smithi088_462306_object131:head# 2024-04-25T09:10:04.922 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.930 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:11a8d835:::benchmark_data_smithi088_462306_object20:head# 2024-04-25T09:10:04.930 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.938 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:12bf1666:::benchmark_data_smithi088_462306_object63:head# 2024-04-25T09:10:04.938 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.947 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1368f5d1:::benchmark_data_smithi088_462306_object208:head# 2024-04-25T09:10:04.947 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.955 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:13b33233:::benchmark_data_smithi088_462306_object40:head# 2024-04-25T09:10:04.955 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.964 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:140e1661:::benchmark_data_smithi088_462306_object247:head# 2024-04-25T09:10:04.964 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.972 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:151d580a:::benchmark_data_smithi088_462306_object59:head# 2024-04-25T09:10:04.972 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.980 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:16cc41f5:::benchmark_data_smithi088_462306_object67:head# 2024-04-25T09:10:04.980 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.989 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17a2c91b:::benchmark_data_smithi088_462306_object17:head# 2024-04-25T09:10:04.989 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:04.997 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17f6836e:::benchmark_data_smithi088_462306_object256:head# 2024-04-25T09:10:04.997 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.005 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:186d6a7e:::benchmark_data_smithi088_462306_object0:head# 2024-04-25T09:10:05.006 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.014 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:19967450:::benchmark_data_smithi088_462306_object123:head# 2024-04-25T09:10:05.014 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.022 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1a55af78:::benchmark_data_smithi088_462306_object75:head# 2024-04-25T09:10:05.022 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.030 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1acd4c2d:::benchmark_data_smithi088_462306_object187:head# 2024-04-25T09:10:05.030 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.039 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1b6403e6:::benchmark_data_smithi088_462306_object33:head# 2024-04-25T09:10:05.039 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.047 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1bce90c6:::benchmark_data_smithi088_462306_object217:head# 2024-04-25T09:10:05.047 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.055 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1cb6ac0c:::benchmark_data_smithi088_462306_object27:head# 2024-04-25T09:10:05.055 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.064 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1cc79080:::benchmark_data_smithi088_462306_object323:head# 2024-04-25T09:10:05.064 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.072 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1d5218b3:::benchmark_data_smithi088_462306_object196:head# 2024-04-25T09:10:05.072 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.080 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1e94d3aa:::benchmark_data_smithi088_462306_object111:head# 2024-04-25T09:10:05.081 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.089 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1eddebf7:::benchmark_data_smithi088_462306_object264:head# 2024-04-25T09:10:05.089 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.097 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1f00585f:::benchmark_data_smithi088_462306_object236:head# 2024-04-25T09:10:05.097 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.105 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20808529:::benchmark_data_smithi088_462306_object151:head# 2024-04-25T09:10:05.106 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.114 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20bdd9c8:::benchmark_data_smithi088_462306_object235:head# 2024-04-25T09:10:05.114 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.122 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20edab27:::benchmark_data_smithi088_462306_object164:head# 2024-04-25T09:10:05.123 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.164 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2160d1fc:::benchmark_data_smithi088_462306_object320:head# 2024-04-25T09:10:05.164 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.172 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:220a8d5a:::benchmark_data_smithi088_462306_object77:head# 2024-04-25T09:10:05.172 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.181 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:231dff78:::benchmark_data_smithi088_462306_object239:head# 2024-04-25T09:10:05.181 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.189 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2330fe3c:::benchmark_data_smithi088_462306_object215:head# 2024-04-25T09:10:05.189 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.197 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2337b927:::benchmark_data_smithi088_462306_object191:head# 2024-04-25T09:10:05.197 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.206 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:23a73227:::benchmark_data_smithi088_462306_object212:head# 2024-04-25T09:10:05.206 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.214 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2448079b:::benchmark_data_smithi088_462306_object70:head# 2024-04-25T09:10:05.214 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.222 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:27c74952:::benchmark_data_smithi088_462306_object288:head# 2024-04-25T09:10:05.222 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.231 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:283568ed:::benchmark_data_smithi088_462306_object315:head# 2024-04-25T09:10:05.231 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.239 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:29974a65:::benchmark_data_smithi088_462306_object2:head# 2024-04-25T09:10:05.239 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.247 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2bbd0b7d:::benchmark_data_smithi088_462306_object224:head# 2024-04-25T09:10:05.247 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.256 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f341493:::benchmark_data_smithi088_462306_object96:head# 2024-04-25T09:10:05.256 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.264 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f3e4e35:::benchmark_data_smithi088_462306_object94:head# 2024-04-25T09:10:05.264 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.272 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3089f82a:::benchmark_data_smithi088_462306_object270:head# 2024-04-25T09:10:05.273 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.281 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:30bc55cf:::benchmark_data_smithi088_462306_object179:head# 2024-04-25T09:10:05.281 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.306 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:30ee7b76:::benchmark_data_smithi088_462306_object211:head# 2024-04-25T09:10:05.306 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.314 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:324b040a:::benchmark_data_smithi088_462306_object28:head# 2024-04-25T09:10:05.314 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.323 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:33bdd052:::benchmark_data_smithi088_462306_object185:head# 2024-04-25T09:10:05.323 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.331 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:34a362cf:::benchmark_data_smithi088_462306_object120:head# 2024-04-25T09:10:05.331 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.339 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:36ae6861:::benchmark_data_smithi088_462306_object125:head# 2024-04-25T09:10:05.340 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.348 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:37054d0f:::benchmark_data_smithi088_462306_object194:head# 2024-04-25T09:10:05.348 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.356 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:37196b5a:::benchmark_data_smithi088_462306_object32:head# 2024-04-25T09:10:05.356 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.364 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3a0087c7:::benchmark_data_smithi088_462306_object299:head# 2024-04-25T09:10:05.364 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.373 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3bfc2bf9:::benchmark_data_smithi088_462306_object25:head# 2024-04-25T09:10:05.373 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.381 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3c658397:::benchmark_data_smithi088_462306_object114:head# 2024-04-25T09:10:05.381 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.390 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3ca6aa4a:::benchmark_data_smithi088_462306_object292:head# 2024-04-25T09:10:05.390 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.398 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3df4cae2:::benchmark_data_smithi088_462306_object330:head# 2024-04-25T09:10:05.398 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.406 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3e25b978:::benchmark_data_smithi088_462306_object250:head# 2024-04-25T09:10:05.406 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.417 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3e918cd5:::benchmark_data_smithi088_462306_object287:head# 2024-04-25T09:10:05.417 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.423 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3f11fd9d:::benchmark_data_smithi088_462306_object127:head# 2024-04-25T09:10:05.423 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.431 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3f85196b:::benchmark_data_smithi088_462306_object135:head# 2024-04-25T09:10:05.431 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.439 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:41e0f85b:::benchmark_data_smithi088_462306_object200:head# 2024-04-25T09:10:05.440 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.448 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:43088259:::benchmark_data_smithi088_462306_object193:head# 2024-04-25T09:10:05.448 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.456 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:452dfc2d:::benchmark_data_smithi088_462306_object144:head# 2024-04-25T09:10:05.456 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.465 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46484682:::benchmark_data_smithi088_462306_object324:head# 2024-04-25T09:10:05.465 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.473 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46c66152:::benchmark_data_smithi088_462306_object149:head# 2024-04-25T09:10:05.473 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.481 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46f14c22:::benchmark_data_smithi088_462306_object300:head# 2024-04-25T09:10:05.481 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.490 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4b72756f:::benchmark_data_smithi088_462306_object329:head# 2024-04-25T09:10:05.490 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.498 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4ba5396c:::benchmark_data_smithi088_462306_object71:head# 2024-04-25T09:10:05.498 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.506 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c34ed72:::benchmark_data_smithi088_462306_object154:head# 2024-04-25T09:10:05.506 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.523 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c3e8c07:::benchmark_data_smithi088_462306_object221:head# 2024-04-25T09:10:05.523 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.532 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c6252f1:::benchmark_data_smithi088_462306_object18:head# 2024-04-25T09:10:05.532 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.540 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4cf34651:::benchmark_data_smithi088_462306_object188:head# 2024-04-25T09:10:05.540 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.548 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d14555f:::benchmark_data_smithi088_462306_object121:head# 2024-04-25T09:10:05.548 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.557 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d5ba4b9:::benchmark_data_smithi088_462306_object158:head# 2024-04-25T09:10:05.557 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.565 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d7ce59d:::benchmark_data_smithi088_462306_object104:head# 2024-04-25T09:10:05.565 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.573 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e3dbe21:::benchmark_data_smithi088_462306_object50:head# 2024-04-25T09:10:05.574 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.582 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e4657de:::benchmark_data_smithi088_462306_object73:head# 2024-04-25T09:10:05.582 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.590 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e9de313:::benchmark_data_smithi088_462306_object93:head# 2024-04-25T09:10:05.590 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.598 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:50f6d9da:::benchmark_data_smithi088_462306_object307:head# 2024-04-25T09:10:05.598 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.607 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:52f10993:::benchmark_data_smithi088_462306_object205:head# 2024-04-25T09:10:05.607 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.615 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:539a6d00:::benchmark_data_smithi088_462306_object107:head# 2024-04-25T09:10:05.615 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.623 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:549c3943:::benchmark_data_smithi088_462306_object23:head# 2024-04-25T09:10:05.624 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.632 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:55901106:::benchmark_data_smithi088_462306_object89:head# 2024-04-25T09:10:05.632 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.640 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:56f25dc9:::benchmark_data_smithi088_462306_object68:head# 2024-04-25T09:10:05.640 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.649 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:56f2d5c4:::benchmark_data_smithi088_462306_object274:head# 2024-04-25T09:10:05.649 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.657 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:58ffc1f4:::benchmark_data_smithi088_462306_object124:head# 2024-04-25T09:10:05.657 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.665 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:59107f03:::benchmark_data_smithi088_462306_object157:head# 2024-04-25T09:10:05.665 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.674 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5933a9b4:::benchmark_data_smithi088_462306_object4:head# 2024-04-25T09:10:05.674 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.682 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:594172d7:::benchmark_data_smithi088_462306_object95:head# 2024-04-25T09:10:05.682 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.690 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5abcca99:::benchmark_data_smithi088_462306_object85:head# 2024-04-25T09:10:05.690 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.699 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5b39136e:::benchmark_data_smithi088_462306_object168:head# 2024-04-25T09:10:05.699 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.707 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5bbb1972:::benchmark_data_smithi088_462306_object113:head# 2024-04-25T09:10:05.707 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.715 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c0a9a1b:::benchmark_data_smithi088_462306_object80:head# 2024-04-25T09:10:05.715 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.724 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c15d000:::benchmark_data_smithi088_462306_object318:head# 2024-04-25T09:10:05.724 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.732 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c895488:::benchmark_data_smithi088_462306_object58:head# 2024-04-25T09:10:05.732 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.740 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5cffb09b:::benchmark_data_smithi088_462306_object31:head# 2024-04-25T09:10:05.741 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.749 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5daa5792:::benchmark_data_smithi088_462306_object192:head# 2024-04-25T09:10:05.749 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.757 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5e669d4b:::benchmark_data_smithi088_462306_object248:head# 2024-04-25T09:10:05.757 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.765 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5e753d70:::benchmark_data_smithi088_462306_object88:head# 2024-04-25T09:10:05.766 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.774 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5fc0d91a:::benchmark_data_smithi088_462306_object83:head# 2024-04-25T09:10:05.774 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.782 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:60c47461:::benchmark_data_smithi088_462306_object201:head# 2024-04-25T09:10:05.782 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.791 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62524425:::benchmark_data_smithi088_462306_object284:head# 2024-04-25T09:10:05.791 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.799 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62e454a3:::benchmark_data_smithi088_462306_object10:head# 2024-04-25T09:10:05.799 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.807 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62e744cb:::benchmark_data_smithi088_462306_object297:head# 2024-04-25T09:10:05.807 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.816 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:63a33891:::benchmark_data_smithi088_462306_object102:head# 2024-04-25T09:10:05.816 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.824 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:64acde59:::benchmark_data_smithi088_462306_object321:head# 2024-04-25T09:10:05.824 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.832 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:64ca7391:::benchmark_data_smithi088_462306_object213:head# 2024-04-25T09:10:05.832 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.841 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:65bc1817:::benchmark_data_smithi088_462306_object303:head# 2024-04-25T09:10:05.841 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.891 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6633f92d:::benchmark_data_smithi088_462306_object126:head# 2024-04-25T09:10:05.891 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.899 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:678dbd08:::benchmark_data_smithi088_462306_object52:head# 2024-04-25T09:10:05.899 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.908 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67a6186b:::benchmark_data_smithi088_462306_object327:head# 2024-04-25T09:10:05.908 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.951 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67aa77b1:::benchmark_data_smithi088_462306_object44:head# 2024-04-25T09:10:05.952 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.982 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6806c11f:::benchmark_data_smithi088_462306_object261:head# 2024-04-25T09:10:05.982 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.991 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:68b0f3d0:::benchmark_data_smithi088_462306_object328:head# 2024-04-25T09:10:05.991 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:05.999 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6a010110:::benchmark_data_smithi088_462306_object296:head# 2024-04-25T09:10:05.999 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.007 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6b5538cb:::benchmark_data_smithi088_462306_object308:head# 2024-04-25T09:10:06.008 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.016 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6b7b59b0:::benchmark_data_smithi088_462306_object218:head# 2024-04-25T09:10:06.016 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.024 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6bbd025e:::benchmark_data_smithi088_462306_object141:head# 2024-04-25T09:10:06.024 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.033 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6c16e74e:::benchmark_data_smithi088_462306_object241:head# 2024-04-25T09:10:06.033 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.041 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6d493727:::benchmark_data_smithi088_462306_object86:head# 2024-04-25T09:10:06.041 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.049 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7041d2eb:::benchmark_data_smithi088_462306_object171:head# 2024-04-25T09:10:06.049 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.058 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:72ab0865:::benchmark_data_smithi088_462306_object42:head# 2024-04-25T09:10:06.058 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.066 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:73e18593:::benchmark_data_smithi088_462306_object234:head# 2024-04-25T09:10:06.066 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.074 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7411cf6e:::benchmark_data_smithi088_462306_object305:head# 2024-04-25T09:10:06.075 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.083 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:749c107b:::benchmark_data_smithi088_462306_object29:head# 2024-04-25T09:10:06.083 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.091 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7759d088:::benchmark_data_smithi088_462306_object310:head# 2024-04-25T09:10:06.091 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.099 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:777330ea:::benchmark_data_smithi088_462306_object262:head# 2024-04-25T09:10:06.099 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.108 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:77982bca:::benchmark_data_smithi088_462306_object269:head# 2024-04-25T09:10:06.108 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.116 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7879b0d4:::benchmark_data_smithi088_462306_object313:head# 2024-04-25T09:10:06.116 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.124 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7c39dbd4:::benchmark_data_smithi088_462306_object45:head# 2024-04-25T09:10:06.125 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.133 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7cb590f7:::benchmark_data_smithi088_462306_object177:head# 2024-04-25T09:10:06.133 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.141 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7d29cf47:::benchmark_data_smithi088_462306_object249:head# 2024-04-25T09:10:06.141 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.149 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7db41ef9:::benchmark_data_smithi088_462306_object190:head# 2024-04-25T09:10:06.150 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.158 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7de60352:::benchmark_data_smithi088_462306_object210:head# 2024-04-25T09:10:06.158 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.166 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7e59f398:::benchmark_data_smithi088_462306_object110:head# 2024-04-25T09:10:06.166 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.174 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7efa7304:::benchmark_data_smithi088_462306_object74:head# 2024-04-25T09:10:06.175 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.183 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7fd54788:::benchmark_data_smithi088_462306_object133:head# 2024-04-25T09:10:06.183 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:06.191 INFO:tasks.workunit.client.0.smithi088.stdout:write_pg epoch 8 info 1.0( v 8'158 (0'0,8'158] local-lis/les=6/7 n=158 ec=6/6 lis/c=6/6 les/c/f=7/7/0 sis=6) 2024-04-25T09:10:06.208 INFO:tasks.workunit.client.0.smithi088.stdout:Import successful 2024-04-25T09:10:06.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:151: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:10:07.459 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T09:10:07.459 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T09:10:07.459 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:10:07.525 INFO:tasks.workunit.client.0.smithi088.stdout:Remove successful 2024-04-25T09:10:08.004 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:152: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-04-25T09:10:08.704 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T09:10:08.704 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T09:10:08.704 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:10:08.817 INFO:tasks.workunit.client.0.smithi088.stdout:Remove successful 2024-04-25T09:10:09.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:154: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0.later --force 2024-04-25T09:10:09.884 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e227 pg_nums {1={18=1}} deleted_pools ) 2024-04-25T09:10:09.884 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.0 2024-04-25T09:10:09.885 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:004ab2c0:::benchmark_data_smithi088_467817_object117:head# 2024-04-25T09:10:09.885 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0054a4c5:::benchmark_data_smithi088_462306_object56:head# 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.009 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0091abb9:::benchmark_data_smithi088_462306_object216:head# 2024-04-25T09:10:10.009 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.042 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:01a57ad0:::benchmark_data_smithi088_462306_object317:head# 2024-04-25T09:10:10.042 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.067 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:01e683c2:::benchmark_data_smithi088_462306_object106:head# 2024-04-25T09:10:10.067 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.084 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0206297e:::benchmark_data_smithi088_467817_object18:head# 2024-04-25T09:10:10.084 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.100 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:021a8ccb:::benchmark_data_smithi088_462306_object108:head# 2024-04-25T09:10:10.100 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.118 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:022d1a7b:::benchmark_data_smithi088_462306_object223:head# 2024-04-25T09:10:10.118 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.134 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0265fcfd:::benchmark_data_smithi088_467817_object94:head# 2024-04-25T09:10:10.134 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.150 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0496a541:::benchmark_data_smithi088_462306_object115:head# 2024-04-25T09:10:10.151 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.167 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:052189ea:::benchmark_data_smithi088_462306_object198:head# 2024-04-25T09:10:10.167 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.184 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:052d5fd4:::benchmark_data_smithi088_462306_object14:head# 2024-04-25T09:10:10.184 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.209 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0589e269:::benchmark_data_smithi088_467817_object98:head# 2024-04-25T09:10:10.209 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.226 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:05f37ee5:::benchmark_data_smithi088_467817_object19:head# 2024-04-25T09:10:10.226 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.242 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:06462b6c:::benchmark_data_smithi088_462306_object237:head# 2024-04-25T09:10:10.242 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.259 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0665cd86:::benchmark_data_smithi088_462306_object266:head# 2024-04-25T09:10:10.259 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.276 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:072ee852:::benchmark_data_smithi088_467817_object116:head# 2024-04-25T09:10:10.276 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.292 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:089ed0f9:::benchmark_data_smithi088_467817_object47:head# 2024-04-25T09:10:10.292 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.309 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:08ef67f6:::benchmark_data_smithi088_462306_object167:head# 2024-04-25T09:10:10.309 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.326 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:09033334:::benchmark_data_smithi088_462306_object57:head# 2024-04-25T09:10:10.326 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.342 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:09cc47f1:::benchmark_data_smithi088_467817_object90:head# 2024-04-25T09:10:10.342 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.359 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a07ee66:::benchmark_data_smithi088_462306_object139:head# 2024-04-25T09:10:10.359 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.376 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a9dbfd3:::benchmark_data_smithi088_462306_object147:head# 2024-04-25T09:10:10.376 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.412 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b31df74:::benchmark_data_smithi088_462306_object103:head# 2024-04-25T09:10:10.412 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.463 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b343d83:::benchmark_data_smithi088_467817_object8:head# 2024-04-25T09:10:10.463 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.513 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b35794e:::benchmark_data_smithi088_462306_object309:head# 2024-04-25T09:10:10.513 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.551 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b54d91a:::benchmark_data_smithi088_462306_object143:head# 2024-04-25T09:10:10.551 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.568 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0b783152:::benchmark_data_smithi088_462306_object53:head# 2024-04-25T09:10:10.568 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.584 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0be5222f:::benchmark_data_smithi088_462306_object295:head# 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.601 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0c2e7645:::benchmark_data_smithi088_462306_object65:head# 2024-04-25T09:10:10.601 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.618 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0d3bf401:::benchmark_data_smithi088_462306_object84:head# 2024-04-25T09:10:10.618 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.634 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0da5c762:::benchmark_data_smithi088_462306_object137:head# 2024-04-25T09:10:10.634 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.651 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0efc8997:::benchmark_data_smithi088_462306_object228:head# 2024-04-25T09:10:10.651 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.668 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0f3cc8d1:::benchmark_data_smithi088_462306_object105:head# 2024-04-25T09:10:10.668 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.684 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0f6ddd44:::benchmark_data_smithi088_462306_object131:head# 2024-04-25T09:10:10.684 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.701 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1039c45c:::benchmark_data_smithi088_467817_object33:head# 2024-04-25T09:10:10.701 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.718 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:10ea7e0c:::benchmark_data_smithi088_467817_object31:head# 2024-04-25T09:10:10.718 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.734 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:11a8d835:::benchmark_data_smithi088_462306_object20:head# 2024-04-25T09:10:10.735 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.751 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:12bd0b48:::benchmark_data_smithi088_467817_object83:head# 2024-04-25T09:10:10.751 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.768 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:12bf1666:::benchmark_data_smithi088_462306_object63:head# 2024-04-25T09:10:10.768 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.784 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1368f5d1:::benchmark_data_smithi088_462306_object208:head# 2024-04-25T09:10:10.785 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.801 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:13b33233:::benchmark_data_smithi088_462306_object40:head# 2024-04-25T09:10:10.801 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.818 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:13c66d49:::benchmark_data_smithi088_467817_object46:head# 2024-04-25T09:10:10.818 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.834 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:140e1661:::benchmark_data_smithi088_462306_object247:head# 2024-04-25T09:10:10.835 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.851 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:148e010e:::benchmark_data_smithi088_467817_object25:head# 2024-04-25T09:10:10.851 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.868 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:151d580a:::benchmark_data_smithi088_462306_object59:head# 2024-04-25T09:10:10.868 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.884 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:15cce5ab:::benchmark_data_smithi088_467817_object126:head# 2024-04-25T09:10:10.885 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.901 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:16cc41f5:::benchmark_data_smithi088_462306_object67:head# 2024-04-25T09:10:10.901 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.918 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17a2c91b:::benchmark_data_smithi088_462306_object17:head# 2024-04-25T09:10:10.918 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.935 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17f6836e:::benchmark_data_smithi088_462306_object256:head# 2024-04-25T09:10:10.935 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.951 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:186d6a7e:::benchmark_data_smithi088_462306_object0:head# 2024-04-25T09:10:10.951 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.968 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:195aac3a:::benchmark_data_smithi088_467817_object88:head# 2024-04-25T09:10:10.968 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:10.985 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:19967450:::benchmark_data_smithi088_462306_object123:head# 2024-04-25T09:10:10.985 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.001 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1a55af78:::benchmark_data_smithi088_462306_object75:head# 2024-04-25T09:10:11.002 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.018 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1a90b07a:::benchmark_data_smithi088_467817_object40:head# 2024-04-25T09:10:11.018 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.035 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1acd4c2d:::benchmark_data_smithi088_462306_object187:head# 2024-04-25T09:10:11.035 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.051 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1b6403e6:::benchmark_data_smithi088_462306_object33:head# 2024-04-25T09:10:11.051 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.068 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1bce90c6:::benchmark_data_smithi088_462306_object217:head# 2024-04-25T09:10:11.068 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.085 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1cb6ac0c:::benchmark_data_smithi088_462306_object27:head# 2024-04-25T09:10:11.085 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.101 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1cc79080:::benchmark_data_smithi088_462306_object323:head# 2024-04-25T09:10:11.102 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.118 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1d5218b3:::benchmark_data_smithi088_462306_object196:head# 2024-04-25T09:10:11.118 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.135 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1e8c7435:::benchmark_data_smithi088_467817_object48:head# 2024-04-25T09:10:11.135 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.151 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1e94d3aa:::benchmark_data_smithi088_462306_object111:head# 2024-04-25T09:10:11.152 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.168 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1eddebf7:::benchmark_data_smithi088_462306_object264:head# 2024-04-25T09:10:11.168 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.185 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1f00585f:::benchmark_data_smithi088_462306_object236:head# 2024-04-25T09:10:11.185 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.235 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1f02ec56:::benchmark_data_smithi088_467817_object89:head# 2024-04-25T09:10:11.235 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.243 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1fd6b57c:::benchmark_data_smithi088_467817_object62:head# 2024-04-25T09:10:11.243 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.252 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20808529:::benchmark_data_smithi088_462306_object151:head# 2024-04-25T09:10:11.252 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.268 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20bdd9c8:::benchmark_data_smithi088_462306_object235:head# 2024-04-25T09:10:11.268 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.285 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:20edab27:::benchmark_data_smithi088_462306_object164:head# 2024-04-25T09:10:11.285 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.293 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:214dd915:::benchmark_data_smithi088_467817_object85:head# 2024-04-25T09:10:11.293 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.302 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2160d1fc:::benchmark_data_smithi088_462306_object320:head# 2024-04-25T09:10:11.302 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.310 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:220a8d5a:::benchmark_data_smithi088_462306_object77:head# 2024-04-25T09:10:11.310 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.318 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:225cea0d:::benchmark_data_smithi088_467817_object75:head# 2024-04-25T09:10:11.318 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.327 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:22cdb4aa:::benchmark_data_smithi088_467817_object84:head# 2024-04-25T09:10:11.327 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.335 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:231dff78:::benchmark_data_smithi088_462306_object239:head# 2024-04-25T09:10:11.335 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.343 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2330fe3c:::benchmark_data_smithi088_462306_object215:head# 2024-04-25T09:10:11.344 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.352 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2337b927:::benchmark_data_smithi088_462306_object191:head# 2024-04-25T09:10:11.352 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.360 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:23a73227:::benchmark_data_smithi088_462306_object212:head# 2024-04-25T09:10:11.360 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.368 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:24024e88:::benchmark_data_smithi088_467817_object101:head# 2024-04-25T09:10:11.369 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.377 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2448079b:::benchmark_data_smithi088_462306_object70:head# 2024-04-25T09:10:11.377 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.385 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:24d502c0:::benchmark_data_smithi088_467817_object99:head# 2024-04-25T09:10:11.385 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.394 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:258537a5:::benchmark_data_smithi088_467817_object112:head# 2024-04-25T09:10:11.394 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.402 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:27359498:::benchmark_data_smithi088_467817_object96:head# 2024-04-25T09:10:11.402 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.410 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:27c74952:::benchmark_data_smithi088_462306_object288:head# 2024-04-25T09:10:11.410 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.419 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:283568ed:::benchmark_data_smithi088_462306_object315:head# 2024-04-25T09:10:11.419 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.427 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:28fdadb1:::benchmark_data_smithi088_467817_object54:head# 2024-04-25T09:10:11.427 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.435 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:28ff6e11:::benchmark_data_smithi088_467817_object27:head# 2024-04-25T09:10:11.435 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.444 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:29974a65:::benchmark_data_smithi088_462306_object2:head# 2024-04-25T09:10:11.444 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.452 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2bbd0b7d:::benchmark_data_smithi088_462306_object224:head# 2024-04-25T09:10:11.452 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.460 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2e92b507:::benchmark_data_smithi088_467817_object37:head# 2024-04-25T09:10:11.461 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.469 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f341493:::benchmark_data_smithi088_462306_object96:head# 2024-04-25T09:10:11.469 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.477 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f3e4e35:::benchmark_data_smithi088_462306_object94:head# 2024-04-25T09:10:11.477 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.485 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3089f82a:::benchmark_data_smithi088_462306_object270:head# 2024-04-25T09:10:11.485 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.494 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:30bc55cf:::benchmark_data_smithi088_462306_object179:head# 2024-04-25T09:10:11.494 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.502 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:30ee7b76:::benchmark_data_smithi088_462306_object211:head# 2024-04-25T09:10:11.502 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.510 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:324b040a:::benchmark_data_smithi088_462306_object28:head# 2024-04-25T09:10:11.510 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.519 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:325722a7:::benchmark_data_smithi088_467817_object124:head# 2024-04-25T09:10:11.519 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.527 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:33bdd052:::benchmark_data_smithi088_462306_object185:head# 2024-04-25T09:10:11.527 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.536 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:34099f88:::benchmark_data_smithi088_467817_object41:head# 2024-04-25T09:10:11.536 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.544 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:34a362cf:::benchmark_data_smithi088_462306_object120:head# 2024-04-25T09:10:11.544 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.552 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:353eace7:::benchmark_data_smithi088_467817_object106:head# 2024-04-25T09:10:11.552 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.561 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3577de39:::benchmark_data_smithi088_467817_object29:head# 2024-04-25T09:10:11.561 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.569 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:36560ed1:::benchmark_data_smithi088_467817_object67:head# 2024-04-25T09:10:11.569 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.577 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:36ae6861:::benchmark_data_smithi088_462306_object125:head# 2024-04-25T09:10:11.578 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.586 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:36f56c74:::benchmark_data_smithi088_467817_object60:head# 2024-04-25T09:10:11.586 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.594 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:37054d0f:::benchmark_data_smithi088_462306_object194:head# 2024-04-25T09:10:11.594 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.602 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:37196b5a:::benchmark_data_smithi088_462306_object32:head# 2024-04-25T09:10:11.602 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.611 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3842e360:::benchmark_data_smithi088_467817_object50:head# 2024-04-25T09:10:11.611 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.619 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3a0087c7:::benchmark_data_smithi088_462306_object299:head# 2024-04-25T09:10:11.619 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.627 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3bfc2bf9:::benchmark_data_smithi088_462306_object25:head# 2024-04-25T09:10:11.627 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.636 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3c658397:::benchmark_data_smithi088_462306_object114:head# 2024-04-25T09:10:11.636 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.644 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3ca6aa4a:::benchmark_data_smithi088_462306_object292:head# 2024-04-25T09:10:11.644 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.652 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3dab0fd5:::benchmark_data_smithi088_467817_object108:head# 2024-04-25T09:10:11.653 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.661 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3db8775a:::benchmark_data_smithi088_467817_object55:head# 2024-04-25T09:10:11.661 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.669 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3df4cae2:::benchmark_data_smithi088_462306_object330:head# 2024-04-25T09:10:11.669 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.678 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3e25b978:::benchmark_data_smithi088_462306_object250:head# 2024-04-25T09:10:11.678 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.686 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3e918cd5:::benchmark_data_smithi088_462306_object287:head# 2024-04-25T09:10:11.686 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.694 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3f11fd9d:::benchmark_data_smithi088_462306_object127:head# 2024-04-25T09:10:11.694 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.703 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3f85196b:::benchmark_data_smithi088_462306_object135:head# 2024-04-25T09:10:11.703 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.711 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:41e0f85b:::benchmark_data_smithi088_462306_object200:head# 2024-04-25T09:10:11.711 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.719 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:43088259:::benchmark_data_smithi088_462306_object193:head# 2024-04-25T09:10:11.719 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.728 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4359c343:::benchmark_data_smithi088_467817_object76:head# 2024-04-25T09:10:11.728 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.736 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:452dfc2d:::benchmark_data_smithi088_462306_object144:head# 2024-04-25T09:10:11.736 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.744 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46484682:::benchmark_data_smithi088_462306_object324:head# 2024-04-25T09:10:11.744 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.753 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46c66152:::benchmark_data_smithi088_462306_object149:head# 2024-04-25T09:10:11.753 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.761 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:46f14c22:::benchmark_data_smithi088_462306_object300:head# 2024-04-25T09:10:11.761 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.769 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:496c1623:::benchmark_data_smithi088_467817_object22:head# 2024-04-25T09:10:11.769 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.778 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4985fbff:::benchmark_data_smithi088_467817_object68:head# 2024-04-25T09:10:11.778 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.828 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:49e3049a:::benchmark_data_smithi088_467817_object79:head# 2024-04-25T09:10:11.828 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.836 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4b72756f:::benchmark_data_smithi088_462306_object329:head# 2024-04-25T09:10:11.836 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.845 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4ba5396c:::benchmark_data_smithi088_462306_object71:head# 2024-04-25T09:10:11.845 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.861 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c34ed72:::benchmark_data_smithi088_462306_object154:head# 2024-04-25T09:10:11.861 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.869 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c3e8c07:::benchmark_data_smithi088_462306_object221:head# 2024-04-25T09:10:11.870 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.878 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c6252f1:::benchmark_data_smithi088_462306_object18:head# 2024-04-25T09:10:11.878 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.886 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4cf34651:::benchmark_data_smithi088_462306_object188:head# 2024-04-25T09:10:11.886 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.895 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d14555f:::benchmark_data_smithi088_462306_object121:head# 2024-04-25T09:10:11.895 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.903 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d5ba4b9:::benchmark_data_smithi088_462306_object158:head# 2024-04-25T09:10:11.903 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.911 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4d7ce59d:::benchmark_data_smithi088_462306_object104:head# 2024-04-25T09:10:11.911 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.959 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e3dbe21:::benchmark_data_smithi088_462306_object50:head# 2024-04-25T09:10:11.959 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:11.995 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e4657de:::benchmark_data_smithi088_462306_object73:head# 2024-04-25T09:10:11.995 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.003 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4e9de313:::benchmark_data_smithi088_462306_object93:head# 2024-04-25T09:10:12.003 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.011 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4ecc8c89:::benchmark_data_smithi088_467817_object71:head# 2024-04-25T09:10:12.011 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.020 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:502a7648:::benchmark_data_smithi088_467817_object118:head# 2024-04-25T09:10:12.020 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.028 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:507e74a0:::benchmark_data_smithi088_467817_object109:head# 2024-04-25T09:10:12.028 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.036 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:50f6d9da:::benchmark_data_smithi088_462306_object307:head# 2024-04-25T09:10:12.036 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.045 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:52f10993:::benchmark_data_smithi088_462306_object205:head# 2024-04-25T09:10:12.045 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.053 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:539a6d00:::benchmark_data_smithi088_462306_object107:head# 2024-04-25T09:10:12.053 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.061 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:549c3943:::benchmark_data_smithi088_462306_object23:head# 2024-04-25T09:10:12.062 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.070 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:55901106:::benchmark_data_smithi088_462306_object89:head# 2024-04-25T09:10:12.070 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.078 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:56f25dc9:::benchmark_data_smithi088_462306_object68:head# 2024-04-25T09:10:12.078 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.087 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:56f2d5c4:::benchmark_data_smithi088_462306_object274:head# 2024-04-25T09:10:12.087 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.095 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:58ffc1f4:::benchmark_data_smithi088_462306_object124:head# 2024-04-25T09:10:12.095 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.103 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:59107f03:::benchmark_data_smithi088_462306_object157:head# 2024-04-25T09:10:12.103 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.111 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:59287275:::benchmark_data_smithi088_467817_object17:head# 2024-04-25T09:10:12.112 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.120 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5933a9b4:::benchmark_data_smithi088_462306_object4:head# 2024-04-25T09:10:12.120 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.128 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:594172d7:::benchmark_data_smithi088_462306_object95:head# 2024-04-25T09:10:12.128 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.136 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:59d8cbe5:::benchmark_data_smithi088_467817_object3:head# 2024-04-25T09:10:12.137 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.187 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5abcca99:::benchmark_data_smithi088_462306_object85:head# 2024-04-25T09:10:12.187 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.220 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5b39136e:::benchmark_data_smithi088_462306_object168:head# 2024-04-25T09:10:12.220 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.264 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5bbb1972:::benchmark_data_smithi088_462306_object113:head# 2024-04-25T09:10:12.264 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.303 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c0a9a1b:::benchmark_data_smithi088_462306_object80:head# 2024-04-25T09:10:12.303 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.370 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c15d000:::benchmark_data_smithi088_462306_object318:head# 2024-04-25T09:10:12.371 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.378 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5c895488:::benchmark_data_smithi088_462306_object58:head# 2024-04-25T09:10:12.378 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.431 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5cbbf528:::benchmark_data_smithi088_467817_object127:head# 2024-04-25T09:10:12.431 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.498 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5cffb09b:::benchmark_data_smithi088_462306_object31:head# 2024-04-25T09:10:12.498 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.560 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5daa5792:::benchmark_data_smithi088_462306_object192:head# 2024-04-25T09:10:12.560 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.606 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5e2ada91:::benchmark_data_smithi088_467817_object32:head# 2024-04-25T09:10:12.606 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.620 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5e669d4b:::benchmark_data_smithi088_462306_object248:head# 2024-04-25T09:10:12.620 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.628 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5e753d70:::benchmark_data_smithi088_462306_object88:head# 2024-04-25T09:10:12.628 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.637 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5f00c71b:::benchmark_data_smithi088_467817_object7:head# 2024-04-25T09:10:12.637 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.645 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5fc0d91a:::benchmark_data_smithi088_462306_object83:head# 2024-04-25T09:10:12.645 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.653 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:60c47461:::benchmark_data_smithi088_462306_object201:head# 2024-04-25T09:10:12.654 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.662 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6116e046:::benchmark_data_smithi088_467817_object87:head# 2024-04-25T09:10:12.662 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.670 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62524425:::benchmark_data_smithi088_462306_object284:head# 2024-04-25T09:10:12.670 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.678 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6299f2a8:::benchmark_data_smithi088_467817_object97:head# 2024-04-25T09:10:12.679 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.687 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62e454a3:::benchmark_data_smithi088_462306_object10:head# 2024-04-25T09:10:12.687 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.695 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62e744cb:::benchmark_data_smithi088_462306_object297:head# 2024-04-25T09:10:12.695 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.703 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:637a868d:::benchmark_data_smithi088_467817_object0:head# 2024-04-25T09:10:12.704 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.712 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:63a33891:::benchmark_data_smithi088_462306_object102:head# 2024-04-25T09:10:12.712 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.720 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:64acde59:::benchmark_data_smithi088_462306_object321:head# 2024-04-25T09:10:12.720 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.729 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:64ca7391:::benchmark_data_smithi088_462306_object213:head# 2024-04-25T09:10:12.729 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.737 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:65bc1817:::benchmark_data_smithi088_462306_object303:head# 2024-04-25T09:10:12.737 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.745 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6633f92d:::benchmark_data_smithi088_462306_object126:head# 2024-04-25T09:10:12.745 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.754 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:678dbd08:::benchmark_data_smithi088_462306_object52:head# 2024-04-25T09:10:12.754 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.762 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67a6186b:::benchmark_data_smithi088_462306_object327:head# 2024-04-25T09:10:12.762 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.770 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67aa77b1:::benchmark_data_smithi088_462306_object44:head# 2024-04-25T09:10:12.770 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.779 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67befc61:::benchmark_data_smithi088_467817_object57:head# 2024-04-25T09:10:12.779 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.787 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6806c11f:::benchmark_data_smithi088_462306_object261:head# 2024-04-25T09:10:12.788 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.795 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:68b0f3d0:::benchmark_data_smithi088_462306_object328:head# 2024-04-25T09:10:12.796 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.804 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6a010110:::benchmark_data_smithi088_462306_object296:head# 2024-04-25T09:10:12.804 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.812 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6a52debd:::benchmark_data_smithi088_467817_object107:head# 2024-04-25T09:10:12.812 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.820 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6b5538cb:::benchmark_data_smithi088_462306_object308:head# 2024-04-25T09:10:12.821 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.870 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6b7b59b0:::benchmark_data_smithi088_462306_object218:head# 2024-04-25T09:10:12.871 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.879 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6bbd025e:::benchmark_data_smithi088_462306_object141:head# 2024-04-25T09:10:12.879 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.887 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6c16e74e:::benchmark_data_smithi088_462306_object241:head# 2024-04-25T09:10:12.888 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.896 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6d493727:::benchmark_data_smithi088_462306_object86:head# 2024-04-25T09:10:12.896 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.904 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6ec178d7:::benchmark_data_smithi088_467817_object4:head# 2024-04-25T09:10:12.904 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.912 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6f1ee242:::benchmark_data_smithi088_467817_object73:head# 2024-04-25T09:10:12.912 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.921 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7041d2eb:::benchmark_data_smithi088_462306_object171:head# 2024-04-25T09:10:12.921 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.929 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:70889470:::benchmark_data_smithi088_467817_object82:head# 2024-04-25T09:10:12.929 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.937 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:70c95171:::benchmark_data_smithi088_467817_object20:head# 2024-04-25T09:10:12.937 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.946 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:71511fb6:::benchmark_data_smithi088_467817_object52:head# 2024-04-25T09:10:12.946 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.954 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7260b26c:::benchmark_data_smithi088_467817_object6:head# 2024-04-25T09:10:12.954 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.962 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:72ab0865:::benchmark_data_smithi088_462306_object42:head# 2024-04-25T09:10:12.962 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.971 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:73e18593:::benchmark_data_smithi088_462306_object234:head# 2024-04-25T09:10:12.971 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.979 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7411cf6e:::benchmark_data_smithi088_462306_object305:head# 2024-04-25T09:10:12.979 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.987 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:749c107b:::benchmark_data_smithi088_462306_object29:head# 2024-04-25T09:10:12.988 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:12.996 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7759d088:::benchmark_data_smithi088_462306_object310:head# 2024-04-25T09:10:12.996 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.004 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:777330ea:::benchmark_data_smithi088_462306_object262:head# 2024-04-25T09:10:13.004 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.013 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:77982bca:::benchmark_data_smithi088_462306_object269:head# 2024-04-25T09:10:13.013 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.021 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:78711619:::benchmark_data_smithi088_467817_object14:head# 2024-04-25T09:10:13.021 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.029 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7879b0d4:::benchmark_data_smithi088_462306_object313:head# 2024-04-25T09:10:13.029 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.038 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7c13826a:::benchmark_data_smithi088_467817_object21:head# 2024-04-25T09:10:13.038 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.046 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7c39dbd4:::benchmark_data_smithi088_462306_object45:head# 2024-04-25T09:10:13.046 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.054 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7cb590f7:::benchmark_data_smithi088_462306_object177:head# 2024-04-25T09:10:13.054 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.063 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7d29cf47:::benchmark_data_smithi088_462306_object249:head# 2024-04-25T09:10:13.063 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.071 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7d6d5f52:::benchmark_data_smithi088_467817_object115:head# 2024-04-25T09:10:13.071 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.079 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7db41ef9:::benchmark_data_smithi088_462306_object190:head# 2024-04-25T09:10:13.079 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.088 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7de60352:::benchmark_data_smithi088_462306_object210:head# 2024-04-25T09:10:13.088 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.096 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7e59f398:::benchmark_data_smithi088_462306_object110:head# 2024-04-25T09:10:13.096 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.104 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7efa7304:::benchmark_data_smithi088_462306_object74:head# 2024-04-25T09:10:13.104 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.113 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7fd54788:::benchmark_data_smithi088_462306_object133:head# 2024-04-25T09:10:13.113 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.121 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81099aed:::benchmark_data_smithi088_462306_object181:head# 2024-04-25T09:10:13.121 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.129 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81ae27f9:::benchmark_data_smithi088_462306_object252:head# 2024-04-25T09:10:13.130 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.138 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81b12e8e:::benchmark_data_smithi088_462306_object161:head# 2024-04-25T09:10:13.138 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.146 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81b4c706:::benchmark_data_smithi088_467817_object58:head# 2024-04-25T09:10:13.146 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.154 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:81e3c822:::benchmark_data_smithi088_462306_object64:head# 2024-04-25T09:10:13.155 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.163 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:821c493f:::benchmark_data_smithi088_467817_object119:head# 2024-04-25T09:10:13.163 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.171 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:82743540:::benchmark_data_smithi088_462306_object276:head# 2024-04-25T09:10:13.171 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.179 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:842cb2a9:::benchmark_data_smithi088_462306_object278:head# 2024-04-25T09:10:13.180 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.188 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:84c54114:::benchmark_data_smithi088_462306_object66:head# 2024-04-25T09:10:13.188 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.196 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:85078920:::benchmark_data_smithi088_462306_object62:head# 2024-04-25T09:10:13.196 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.205 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:85f7c2db:::benchmark_data_smithi088_462306_object326:head# 2024-04-25T09:10:13.205 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.213 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:86a2ceef:::benchmark_data_smithi088_467817_object11:head# 2024-04-25T09:10:13.213 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.221 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8770f696:::benchmark_data_smithi088_462306_object253:head# 2024-04-25T09:10:13.222 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.230 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87b0211e:::benchmark_data_smithi088_462306_object8:head# 2024-04-25T09:10:13.230 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.238 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87cd1abd:::benchmark_data_smithi088_462306_object128:head# 2024-04-25T09:10:13.238 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.246 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87cf274d:::benchmark_data_smithi088_462306_object19:head# 2024-04-25T09:10:13.246 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.255 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87f67271:::benchmark_data_smithi088_462306_object189:head# 2024-04-25T09:10:13.255 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.272 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8802f69d:::benchmark_data_smithi088_467817_object26:head# 2024-04-25T09:10:13.272 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.280 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:88f3d2ff:::benchmark_data_smithi088_462306_object173:head# 2024-04-25T09:10:13.356 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.357 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8a4612ce:::benchmark_data_smithi088_462306_object233:head# 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8a69a4db:::benchmark_data_smithi088_467817_object120:head# 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8aa12c5c:::benchmark_data_smithi088_467817_object104:head# 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8bb6f309:::benchmark_data_smithi088_462306_object132:head# 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8ca22c61:::benchmark_data_smithi088_462306_object220:head# 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8ce82677:::benchmark_data_smithi088_467817_object93:head# 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8cfffb40:::benchmark_data_smithi088_467817_object16:head# 2024-04-25T09:10:13.358 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.359 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8d801fc0:::benchmark_data_smithi088_462306_object207:head# 2024-04-25T09:10:13.359 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.359 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8fbb559d:::benchmark_data_smithi088_462306_object257:head# 2024-04-25T09:10:13.359 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.364 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91623a25:::benchmark_data_smithi088_462306_object138:head# 2024-04-25T09:10:13.364 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.372 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9196f898:::benchmark_data_smithi088_462306_object6:head# 2024-04-25T09:10:13.372 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.381 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91e42837:::benchmark_data_smithi088_462306_object176:head# 2024-04-25T09:10:13.381 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.389 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91e98fd5:::benchmark_data_smithi088_462306_object35:head# 2024-04-25T09:10:13.389 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.397 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92220cfe:::benchmark_data_smithi088_462306_object22:head# 2024-04-25T09:10:13.397 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.405 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9265404e:::benchmark_data_smithi088_462306_object169:head# 2024-04-25T09:10:13.405 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.464 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92ad4e27:::benchmark_data_smithi088_462306_object238:head# 2024-04-25T09:10:13.464 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.472 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:934c91ea:::benchmark_data_smithi088_462306_object246:head# 2024-04-25T09:10:13.472 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.481 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:937e9a49:::benchmark_data_smithi088_462306_object199:head# 2024-04-25T09:10:13.481 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.489 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:93b8c46b:::benchmark_data_smithi088_462306_object76:head# 2024-04-25T09:10:13.489 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.497 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9411c558:::benchmark_data_smithi088_462306_object24:head# 2024-04-25T09:10:13.497 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.506 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:94abe535:::benchmark_data_smithi088_462306_object60:head# 2024-04-25T09:10:13.506 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.514 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:96cf9e3a:::benchmark_data_smithi088_462306_object325:head# 2024-04-25T09:10:13.514 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.522 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:981a16b9:::benchmark_data_smithi088_467817_object53:head# 2024-04-25T09:10:13.522 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.531 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:98888887:::benchmark_data_smithi088_462306_object130:head# 2024-04-25T09:10:13.531 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.539 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:98cc271a:::benchmark_data_smithi088_462306_object174:head# 2024-04-25T09:10:13.539 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.547 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9935b1ed:::benchmark_data_smithi088_462306_object272:head# 2024-04-25T09:10:13.547 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.556 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9938cea0:::benchmark_data_smithi088_462306_object162:head# 2024-04-25T09:10:13.556 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.564 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:99c8e7d5:::benchmark_data_smithi088_467817_object36:head# 2024-04-25T09:10:13.564 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.572 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9acb5316:::benchmark_data_smithi088_462306_object222:head# 2024-04-25T09:10:13.573 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.581 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9b089756:::benchmark_data_smithi088_462306_object271:head# 2024-04-25T09:10:13.581 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.589 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9baeb73f:::benchmark_data_smithi088_462306_object117:head# 2024-04-25T09:10:13.589 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.597 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9bf57a33:::benchmark_data_smithi088_462306_object204:head# 2024-04-25T09:10:13.598 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.606 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9bf7ace7:::benchmark_data_smithi088_467817_object72:head# 2024-04-25T09:10:13.606 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.614 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9c2d402a:::benchmark_data_smithi088_467817_object66:head# 2024-04-25T09:10:13.614 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.623 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9c595d90:::benchmark_data_smithi088_462306_object16:head# 2024-04-25T09:10:13.623 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.631 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9caedf93:::benchmark_data_smithi088_462306_object109:head# 2024-04-25T09:10:13.631 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.639 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9cb158b2:::benchmark_data_smithi088_462306_object150:head# 2024-04-25T09:10:13.640 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.648 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9cb772db:::benchmark_data_smithi088_467817_object9:head# 2024-04-25T09:10:13.648 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.656 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9cc125fe:::benchmark_data_smithi088_467817_object1:head# 2024-04-25T09:10:13.656 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.664 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9cd16357:::benchmark_data_smithi088_467817_object5:head# 2024-04-25T09:10:13.664 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.673 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9f6e371a:::benchmark_data_smithi088_462306_object275:head# 2024-04-25T09:10:13.673 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.681 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a20735fd:::benchmark_data_smithi088_462306_object282:head# 2024-04-25T09:10:13.681 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.689 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a2c8e7c0:::benchmark_data_smithi088_462306_object116:head# 2024-04-25T09:10:13.690 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.698 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a3548fec:::benchmark_data_smithi088_462306_object178:head# 2024-04-25T09:10:13.698 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.706 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a3d97e4f:::benchmark_data_smithi088_462306_object37:head# 2024-04-25T09:10:13.706 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.714 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a40b440b:::benchmark_data_smithi088_462306_object197:head# 2024-04-25T09:10:13.715 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.723 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a4fe7d56:::benchmark_data_smithi088_462306_object92:head# 2024-04-25T09:10:13.723 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.731 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6c17891:::benchmark_data_smithi088_462306_object146:head# 2024-04-25T09:10:13.731 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.739 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6cdc671:::benchmark_data_smithi088_462306_object195:head# 2024-04-25T09:10:13.740 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.748 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6ebe5bf:::benchmark_data_smithi088_462306_object145:head# 2024-04-25T09:10:13.748 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.756 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6ef9d72:::benchmark_data_smithi088_467817_object61:head# 2024-04-25T09:10:13.756 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.765 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a7506a6e:::benchmark_data_smithi088_462306_object122:head# 2024-04-25T09:10:13.765 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.773 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a87a78a1:::benchmark_data_smithi088_462306_object5:head# 2024-04-25T09:10:13.773 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.781 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a87bb1e0:::benchmark_data_smithi088_462306_object263:head# 2024-04-25T09:10:13.781 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.790 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a8de3fc3:::benchmark_data_smithi088_462306_object134:head# 2024-04-25T09:10:13.790 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.798 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a92036b2:::benchmark_data_smithi088_462306_object225:head# 2024-04-25T09:10:13.798 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.806 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:aa84cf26:::benchmark_data_smithi088_462306_object11:head# 2024-04-25T09:10:13.806 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.815 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:aaac40ab:::benchmark_data_smithi088_467817_object125:head# 2024-04-25T09:10:13.815 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.823 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab034bbd:::benchmark_data_smithi088_462306_object232:head# 2024-04-25T09:10:13.823 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.831 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab565d7b:::benchmark_data_smithi088_462306_object302:head# 2024-04-25T09:10:13.831 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.840 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:abc81ff6:::benchmark_data_smithi088_462306_object69:head# 2024-04-25T09:10:13.840 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.848 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ac08c85b:::benchmark_data_smithi088_462306_object61:head# 2024-04-25T09:10:13.848 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.856 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:acecab62:::benchmark_data_smithi088_462306_object219:head# 2024-04-25T09:10:13.856 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.865 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ad2c69f2:::benchmark_data_smithi088_467817_object13:head# 2024-04-25T09:10:13.865 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.873 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ad48ca91:::benchmark_data_smithi088_462306_object209:head# 2024-04-25T09:10:13.873 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.882 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:af94d379:::benchmark_data_smithi088_462306_object298:head# 2024-04-25T09:10:13.882 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.890 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b0688a46:::benchmark_data_smithi088_467817_object51:head# 2024-04-25T09:10:13.890 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.898 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b06befdd:::benchmark_data_smithi088_462306_object156:head# 2024-04-25T09:10:13.899 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.907 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b0911968:::benchmark_data_smithi088_467817_object123:head# 2024-04-25T09:10:13.907 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.940 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b0c2cf35:::benchmark_data_smithi088_462306_object240:head# 2024-04-25T09:10:13.940 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.981 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b0e3af7f:::benchmark_data_smithi088_467817_object12:head# 2024-04-25T09:10:13.982 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.990 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b17d8fef:::benchmark_data_smithi088_462306_object13:head# 2024-04-25T09:10:13.990 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:13.999 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b20bd2ac:::benchmark_data_smithi088_462306_object26:head# 2024-04-25T09:10:14.000 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.007 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b2160c47:::benchmark_data_smithi088_462306_object159:head# 2024-04-25T09:10:14.007 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.015 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b2850783:::benchmark_data_smithi088_467817_object80:head# 2024-04-25T09:10:14.015 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.023 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b2f527bb:::benchmark_data_smithi088_462306_object21:head# 2024-04-25T09:10:14.023 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.032 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b32d563a:::benchmark_data_smithi088_467817_object34:head# 2024-04-25T09:10:14.032 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.040 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b35e628e:::benchmark_data_smithi088_467817_object91:head# 2024-04-25T09:10:14.040 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.048 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b38816db:::benchmark_data_smithi088_462306_object87:head# 2024-04-25T09:10:14.048 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.082 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b39555ce:::benchmark_data_smithi088_462306_object311:head# 2024-04-25T09:10:14.082 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.090 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b3d4574c:::benchmark_data_smithi088_462306_object182:head# 2024-04-25T09:10:14.090 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.099 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b50b3477:::benchmark_data_smithi088_462306_object55:head# 2024-04-25T09:10:14.099 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.107 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b5453a8b:::benchmark_data_smithi088_467817_object59:head# 2024-04-25T09:10:14.107 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.115 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b65ada81:::benchmark_data_smithi088_467817_object43:head# 2024-04-25T09:10:14.115 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.124 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b7cd0d37:::benchmark_data_smithi088_462306_object294:head# 2024-04-25T09:10:14.124 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.132 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b82e2947:::benchmark_data_smithi088_462306_object163:head# 2024-04-25T09:10:14.132 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.140 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:10:14.140 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.165 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b8db97a9:::benchmark_data_smithi088_462306_object165:head# 2024-04-25T09:10:14.165 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.174 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b97bb4a3:::benchmark_data_smithi088_462306_object319:head# 2024-04-25T09:10:14.174 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.182 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba2c6dac:::benchmark_data_smithi088_462306_object251:head# 2024-04-25T09:10:14.182 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.190 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba685cfd:::benchmark_data_smithi088_462306_object99:head# 2024-04-25T09:10:14.190 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.199 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba7c5b40:::benchmark_data_smithi088_462306_object202:head# 2024-04-25T09:10:14.199 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.207 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba8d78f2:::benchmark_data_smithi088_462306_object148:head# 2024-04-25T09:10:14.207 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.215 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:baf95fb7:::benchmark_data_smithi088_462306_object290:head# 2024-04-25T09:10:14.215 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.224 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bca58a03:::benchmark_data_smithi088_462306_object129:head# 2024-04-25T09:10:14.224 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.232 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bd7d893a:::benchmark_data_smithi088_467817_object30:head# 2024-04-25T09:10:14.232 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.240 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bdd50353:::benchmark_data_smithi088_462306_object98:head# 2024-04-25T09:10:14.240 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.257 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:be02a273:::benchmark_data_smithi088_462306_object291:head# 2024-04-25T09:10:14.257 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.274 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bf30e66b:::benchmark_data_smithi088_467817_object121:head# 2024-04-25T09:10:14.274 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.282 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bf8b18a3:::benchmark_data_smithi088_467817_object2:head# 2024-04-25T09:10:14.282 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.291 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bfa07270:::benchmark_data_smithi088_462306_object258:head# 2024-04-25T09:10:14.291 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.299 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bfc2ddbb:::benchmark_data_smithi088_462306_object155:head# 2024-04-25T09:10:14.299 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.307 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c18f632f:::benchmark_data_smithi088_462306_object43:head# 2024-04-25T09:10:14.307 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.315 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c1eedab1:::benchmark_data_smithi088_467817_object28:head# 2024-04-25T09:10:14.315 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.324 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c2f3461e:::benchmark_data_smithi088_462306_object91:head# 2024-04-25T09:10:14.324 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.332 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c3b33d4e:::benchmark_data_smithi088_467817_object23:head# 2024-04-25T09:10:14.332 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.341 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c3bd9ba8:::benchmark_data_smithi088_467817_object105:head# 2024-04-25T09:10:14.341 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.349 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c40c5699:::benchmark_data_smithi088_462306_object280:head# 2024-04-25T09:10:14.349 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.357 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c4517802:::benchmark_data_smithi088_462306_object3:head# 2024-04-25T09:10:14.357 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.366 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c4dec443:::benchmark_data_smithi088_462306_object277:head# 2024-04-25T09:10:14.366 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.374 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c588688f:::benchmark_data_smithi088_462306_object273:head# 2024-04-25T09:10:14.374 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.382 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c5ea4073:::benchmark_data_smithi088_467817_object81:head# 2024-04-25T09:10:14.382 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.440 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c621a2c5:::benchmark_data_smithi088_462306_object51:head# 2024-04-25T09:10:14.440 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.484 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c6b2539b:::benchmark_data_smithi088_462306_object254:head# 2024-04-25T09:10:14.484 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.524 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c8b8d985:::benchmark_data_smithi088_462306_object118:head# 2024-04-25T09:10:14.524 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.541 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c8e1a562:::benchmark_data_smithi088_462306_object245:head# 2024-04-25T09:10:14.541 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.549 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c9f5e653:::benchmark_data_smithi088_467817_object78:head# 2024-04-25T09:10:14.549 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.557 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ca40773e:::benchmark_data_smithi088_467817_object44:head# 2024-04-25T09:10:14.557 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.566 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cace5b80:::benchmark_data_smithi088_467817_object110:head# 2024-04-25T09:10:14.566 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.574 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cad804f9:::benchmark_data_smithi088_462306_object285:head# 2024-04-25T09:10:14.574 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.582 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cae4686f:::benchmark_data_smithi088_467817_object100:head# 2024-04-25T09:10:14.583 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.591 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cb0bb72b:::benchmark_data_smithi088_462306_object268:head# 2024-04-25T09:10:14.591 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.599 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cb9c1d39:::benchmark_data_smithi088_462306_object304:head# 2024-04-25T09:10:14.599 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.610 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cba404d8:::benchmark_data_smithi088_462306_object34:head# 2024-04-25T09:10:14.610 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.616 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cc8e65c8:::benchmark_data_smithi088_462306_object334:head# 2024-04-25T09:10:14.616 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.624 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cd3cc54c:::benchmark_data_smithi088_467817_object64:head# 2024-04-25T09:10:14.624 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.632 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cd75ee52:::benchmark_data_smithi088_462306_object172:head# 2024-04-25T09:10:14.633 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.641 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ce72978b:::benchmark_data_smithi088_462306_object226:head# 2024-04-25T09:10:14.641 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.649 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cebe6f8b:::benchmark_data_smithi088_462306_object242:head# 2024-04-25T09:10:14.649 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.658 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cf135ed8:::benchmark_data_smithi088_462306_object97:head# 2024-04-25T09:10:14.658 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.666 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cf1ea7f9:::benchmark_data_smithi088_462306_object337:head# 2024-04-25T09:10:14.666 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.674 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cfc2a95a:::benchmark_data_smithi088_462306_object119:head# 2024-04-25T09:10:14.675 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.683 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cffde5b4:::benchmark_data_smithi088_467817_object103:head# 2024-04-25T09:10:14.683 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.691 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d06b5209:::benchmark_data_smithi088_462306_object152:head# 2024-04-25T09:10:14.691 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.733 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d0efe327:::benchmark_data_smithi088_462306_object306:head# 2024-04-25T09:10:14.733 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.741 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d1723fce:::benchmark_data_smithi088_467817_object77:head# 2024-04-25T09:10:14.741 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.749 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d182b227:::benchmark_data_smithi088_462306_object100:head# 2024-04-25T09:10:14.750 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.758 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d1977b2c:::benchmark_data_smithi088_462306_object166:head# 2024-04-25T09:10:14.758 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.766 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d1e00a1c:::benchmark_data_smithi088_462306_object48:head# 2024-04-25T09:10:14.766 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.775 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d297305a:::benchmark_data_smithi088_462306_object301:head# 2024-04-25T09:10:14.775 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.783 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2cbd392:::benchmark_data_smithi088_462306_object136:head# 2024-04-25T09:10:14.783 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.791 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2dc1847:::benchmark_data_smithi088_462306_object36:head# 2024-04-25T09:10:14.791 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.800 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d2f0530a:::benchmark_data_smithi088_462306_object331:head# 2024-04-25T09:10:14.800 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.850 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d41367b1:::benchmark_data_smithi088_467817_object86:head# 2024-04-25T09:10:14.850 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.858 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d41d1c70:::benchmark_data_smithi088_462306_object183:head# 2024-04-25T09:10:14.858 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.866 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d42ba1fa:::benchmark_data_smithi088_462306_object38:head# 2024-04-25T09:10:14.867 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.875 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d42ca9ea:::benchmark_data_smithi088_462306_object259:head# 2024-04-25T09:10:14.875 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.883 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d4ccf510:::benchmark_data_smithi088_462306_object206:head# 2024-04-25T09:10:14.883 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.892 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d5ddf04f:::benchmark_data_smithi088_467817_object15:head# 2024-04-25T09:10:14.892 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.900 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d658cee6:::benchmark_data_smithi088_462306_object279:head# 2024-04-25T09:10:14.900 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.908 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d6791640:::benchmark_data_smithi088_462306_object9:head# 2024-04-25T09:10:14.908 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.917 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d69229b2:::benchmark_data_smithi088_462306_object312:head# 2024-04-25T09:10:14.917 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.925 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d746333f:::benchmark_data_smithi088_462306_object1:head# 2024-04-25T09:10:14.925 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.933 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d8820786:::benchmark_data_smithi088_462306_object336:head# 2024-04-25T09:10:14.933 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.941 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d8e6af19:::benchmark_data_smithi088_467817_object45:head# 2024-04-25T09:10:14.942 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.950 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d8e83737:::benchmark_data_smithi088_462306_object203:head# 2024-04-25T09:10:14.950 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.958 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d92e92c0:::benchmark_data_smithi088_462306_object153:head# 2024-04-25T09:10:14.958 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.966 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d9ed16a2:::benchmark_data_smithi088_462306_object244:head# 2024-04-25T09:10:14.967 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.975 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d9f9d37f:::benchmark_data_smithi088_462306_object322:head# 2024-04-25T09:10:14.975 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.983 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:da421750:::benchmark_data_smithi088_462306_object283:head# 2024-04-25T09:10:14.983 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:14.992 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dad3e1cc:::benchmark_data_smithi088_462306_object293:head# 2024-04-25T09:10:14.992 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.000 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dbf914c5:::benchmark_data_smithi088_462306_object78:head# 2024-04-25T09:10:15.000 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.008 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dbfcbc35:::benchmark_data_smithi088_462306_object41:head# 2024-04-25T09:10:15.008 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.017 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dc672535:::benchmark_data_smithi088_462306_object82:head# 2024-04-25T09:10:15.017 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.025 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dc835119:::benchmark_data_smithi088_467817_object74:head# 2024-04-25T09:10:15.025 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.034 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd09b3db:::benchmark_data_smithi088_462306_object12:head# 2024-04-25T09:10:15.034 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.042 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:de8da2f7:::benchmark_data_smithi088_462306_object267:head# 2024-04-25T09:10:15.042 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.050 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:df136223:::benchmark_data_smithi088_462306_object81:head# 2024-04-25T09:10:15.050 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.058 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:df552722:::benchmark_data_smithi088_462306_object54:head# 2024-04-25T09:10:15.058 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.067 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dfd38933:::benchmark_data_smithi088_467817_object70:head# 2024-04-25T09:10:15.067 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.075 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e074f344:::benchmark_data_smithi088_462306_object231:head# 2024-04-25T09:10:15.075 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.083 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0d77fb7:::benchmark_data_smithi088_462306_object229:head# 2024-04-25T09:10:15.084 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.092 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0eb6167:::benchmark_data_smithi088_462306_object101:head# 2024-04-25T09:10:15.092 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.100 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e0fbd74b:::benchmark_data_smithi088_462306_object333:head# 2024-04-25T09:10:15.100 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.109 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e1956e90:::benchmark_data_smithi088_462306_object243:head# 2024-04-25T09:10:15.109 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.117 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e22f836d:::benchmark_data_smithi088_462306_object180:head# 2024-04-25T09:10:15.117 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.125 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e25ba6c9:::benchmark_data_smithi088_462306_object15:head# 2024-04-25T09:10:15.125 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.134 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e326ce95:::benchmark_data_smithi088_467817_object39:head# 2024-04-25T09:10:15.134 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.142 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e344d3d7:::benchmark_data_smithi088_467817_object69:head# 2024-04-25T09:10:15.142 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.150 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e3fbb848:::benchmark_data_smithi088_462306_object30:head# 2024-04-25T09:10:15.150 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.159 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e490c252:::benchmark_data_smithi088_462306_object281:head# 2024-04-25T09:10:15.159 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.167 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e4fb9482:::benchmark_data_smithi088_462306_object260:head# 2024-04-25T09:10:15.167 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.176 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e5649a8a:::benchmark_data_smithi088_467817_object38:head# 2024-04-25T09:10:15.176 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.184 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e5dae559:::benchmark_data_smithi088_462306_object170:head# 2024-04-25T09:10:15.184 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.192 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e665af06:::benchmark_data_smithi088_462306_object49:head# 2024-04-25T09:10:15.192 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.201 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e6cff169:::benchmark_data_smithi088_467817_object92:head# 2024-04-25T09:10:15.201 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.209 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e7b1daf4:::benchmark_data_smithi088_462306_object47:head# 2024-04-25T09:10:15.209 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.217 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e8c06845:::benchmark_data_smithi088_467817_object42:head# 2024-04-25T09:10:15.217 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.225 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e9823a3c:::benchmark_data_smithi088_467817_object56:head# 2024-04-25T09:10:15.226 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.234 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e9919842:::benchmark_data_smithi088_462306_object289:head# 2024-04-25T09:10:15.234 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.242 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ebc23c5b:::benchmark_data_smithi088_462306_object316:head# 2024-04-25T09:10:15.242 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.251 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ebddce23:::benchmark_data_smithi088_462306_object286:head# 2024-04-25T09:10:15.251 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.259 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:eccef000:::benchmark_data_smithi088_467817_object122:head# 2024-04-25T09:10:15.259 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.267 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ed2ff011:::benchmark_data_smithi088_467817_object111:head# 2024-04-25T09:10:15.267 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.276 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ef8b7d88:::benchmark_data_smithi088_467817_object10:head# 2024-04-25T09:10:15.276 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.284 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f01c7538:::benchmark_data_smithi088_462306_object255:head# 2024-04-25T09:10:15.284 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.292 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f03d3cff:::benchmark_data_smithi088_462306_object184:head# 2024-04-25T09:10:15.292 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.301 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f0516783:::benchmark_data_smithi088_467817_object35:head# 2024-04-25T09:10:15.301 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.309 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f07d123f:::benchmark_data_smithi088_467817_object49:head# 2024-04-25T09:10:15.309 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.317 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f0b18d8c:::benchmark_data_smithi088_462306_object7:head# 2024-04-25T09:10:15.318 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.326 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f2319ef1:::benchmark_data_smithi088_462306_object46:head# 2024-04-25T09:10:15.326 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.334 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f27dc48a:::benchmark_data_smithi088_462306_object175:head# 2024-04-25T09:10:15.334 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.342 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f2e14e58:::benchmark_data_smithi088_467817_object102:head# 2024-04-25T09:10:15.343 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.351 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f3cf7b95:::benchmark_data_smithi088_462306_object112:head# 2024-04-25T09:10:15.351 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.359 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f3f2f56d:::benchmark_data_smithi088_467817_object63:head# 2024-04-25T09:10:15.359 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.367 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f40a2c8a:::benchmark_data_smithi088_467817_object65:head# 2024-04-25T09:10:15.368 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.376 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f473732d:::benchmark_data_smithi088_462306_object265:head# 2024-04-25T09:10:15.376 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.409 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f4b5afc3:::benchmark_data_smithi088_467817_object24:head# 2024-04-25T09:10:15.409 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.418 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f50d52f3:::benchmark_data_smithi088_462306_object227:head# 2024-04-25T09:10:15.418 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.426 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f69dfeef:::benchmark_data_smithi088_462306_object186:head# 2024-04-25T09:10:15.427 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.434 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f77a4b03:::benchmark_data_smithi088_467817_object113:head# 2024-04-25T09:10:15.434 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.443 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f7adaaf0:::benchmark_data_smithi088_462306_object214:head# 2024-04-25T09:10:15.443 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.451 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f7c73560:::benchmark_data_smithi088_462306_object332:head# 2024-04-25T09:10:15.451 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.459 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f8160d49:::benchmark_data_smithi088_462306_object314:head# 2024-04-25T09:10:15.459 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.468 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f90ccab7:::benchmark_data_smithi088_462306_object142:head# 2024-04-25T09:10:15.468 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.476 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f91a495a:::benchmark_data_smithi088_462306_object72:head# 2024-04-25T09:10:15.476 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.484 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fabfa17e:::benchmark_data_smithi088_462306_object140:head# 2024-04-25T09:10:15.484 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.493 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fb87786e:::benchmark_data_smithi088_462306_object160:head# 2024-04-25T09:10:15.493 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.501 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fbe0687a:::benchmark_data_smithi088_462306_object230:head# 2024-04-25T09:10:15.501 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.509 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fc115286:::benchmark_data_smithi088_462306_object39:head# 2024-04-25T09:10:15.510 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.518 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fd7dad8e:::benchmark_data_smithi088_467817_object95:head# 2024-04-25T09:10:15.518 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.526 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fe11890a:::benchmark_data_smithi088_467817_object114:head# 2024-04-25T09:10:15.526 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.534 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fe55e075:::benchmark_data_smithi088_462306_object335:head# 2024-04-25T09:10:15.535 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.560 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fefad482:::benchmark_data_smithi088_462306_object79:head# 2024-04-25T09:10:15.560 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.568 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ff2bd568:::benchmark_data_smithi088_462306_object90:head# 2024-04-25T09:10:15.568 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:10:15.577 INFO:tasks.workunit.client.0.smithi088.stdout:write_pg epoch 223 info 1.0( v 223'310 (8'181,223'310] local-lis/les=22/23 n=467 ec=6/6 lis/c=22/22 les/c/f=23/23/0 sis=22) 2024-04-25T09:10:15.593 INFO:tasks.workunit.client.0.smithi088.stdout:Import successful 2024-04-25T09:10:16.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:157: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:10:16.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:10:16.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:10:16.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:10:16.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:10:16.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:10:16.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:16.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:16.138 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:10:16.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:10:16.140 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:10:16.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:10:16.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:10:16.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:10:16.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:10:16.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:10:16.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:10:16.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:10:16.175 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:16.188+0000 7f22b54467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:16.178 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:16.188+0000 7f22b54467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:16.179 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:16.192+0000 7f22b54467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:16.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:10:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:10:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:10:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:10:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:10:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:10:16.624 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:10:16.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:16.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:16.755 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:16.768+0000 7f22b54467c0 -1 Falling back to public interface 2024-04-25T09:10:17.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:18.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:18.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:18.099 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:10:18.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:10:18.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:18.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:18.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:19.282 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:19.296+0000 7f22b54467c0 -1 osd.0 227 log_to_monitors true 2024-04-25T09:10:19.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:19.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:19.529 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:10:19.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:10:19.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:19.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:20.077 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 230 up_thru 22 down_at 228 last_clean_interval [22,227) [v2:127.0.0.1:6802/1198406434,v1:127.0.0.1:6803/1198406434] [v2:127.0.0.1:6804/1198406434,v1:127.0.0.1:6805/1198406434] exists,up 168d882f-4939-4337-8b16-26a6cead1b6d 2024-04-25T09:10:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:10:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:10:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:10:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:159: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:10:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:10:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:10:20.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:10:20.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:10:20.080 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:10:20.080 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:10:20.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:10:20.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:10:20.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:10:20.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:10:20.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:10:20.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:10:20.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:10:20.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:10:20.184 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:10:20.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:10:20.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:10:20.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:20.764 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:10:20.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=987842478083 2024-04-25T09:10:20.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 987842478083 2024-04-25T09:10:20.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-987842478083' 2024-04-25T09:10:20.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:20.965 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-987842478083 2024-04-25T09:10:20.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:20.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:10:20.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-987842478083 2024-04-25T09:10:20.967 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:20.968 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 987842478083 2024-04-25T09:10:20.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=987842478083 2024-04-25T09:10:20.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 987842478083' 2024-04-25T09:10:20.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:21.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 987842478083 2024-04-25T09:10:21.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:10:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:10:22.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:22.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 987842478083 -lt 987842478083 2024-04-25T09:10:22.795 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:10:22.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:22.796 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:23.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:10:23.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:10:23.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:10:23.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:10:23.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:10:23.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:10:23.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:10:23.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:10:23.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:10:23.828 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:10:23.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:23.829 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:24.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:10:24.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:10:24.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:10:24.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:10:24.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:10:24.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:10:24.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:10:24.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:10:24.354 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:10:24.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:10:24.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:10:24.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:10:24.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:10:24.663 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:10:24.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:10:24.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:10:24.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:10:24.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:10:24.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:10:24.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:24.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:10:24.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:10:24.669 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:24.670 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:10:24.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:10:24.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:10:24.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:10:24.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:24.692 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:24.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451260 2024-04-25T09:10:24.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:10:24.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:10:24.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:10:24.694 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:10:24.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:10:24.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:10:24.698 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:10:24.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:10:24.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:10:24.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:10:24.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:10:24.701 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:10:24.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:24.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:10:24.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:10:24.703 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:24.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:10:24.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:10:24.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:10:24.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:10:24.706 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:24.707 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:24.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451260 2024-04-25T09:10:24.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:10:24.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:10:24.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:10:24.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:10:24.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:24.709 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:24.709 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.451260 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_split td/pg-split-merge 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:163: TEST_import_after_split: local dir=td/pg-split-merge 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:165: TEST_import_after_split: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:10:24.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:10:24.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:10:24.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:10:24.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:10:24.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:10:25.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:10:25.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:25.186 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:25.186 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:25.186 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:25.187 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:25.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:10:25.187 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:10:25.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:10:25.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:10:25.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:10:25.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:10:25.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:10:25.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:10:25.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:10:25.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:10:25.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:10:25.419 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:25.419 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:25.419 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:25.419 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451260/ceph-mon.a.asok 2024-04-25T09:10:25.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:10:25.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451260/ceph-mon.a.asok config get fsid 2024-04-25T09:10:25.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:10:25.538 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451260/ceph-mon.a.asok 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:10:25.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451260/ceph-mon.a.asok config get mon_host 2024-04-25T09:10:25.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:166: TEST_import_after_split: run_mgr td/pg-split-merge x 2024-04-25T09:10:25.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:10:25.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:10:25.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:10:25.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:10:25.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:10:25.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:10:26.539 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:10:26.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:26.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:26.540 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:26.540 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:26.540 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:26.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:10:26.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:10:26.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:10:26.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:167: TEST_import_after_split: run_osd td/pg-split-merge 0 2024-04-25T09:10:26.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:10:26.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:10:26.577 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:10:26.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:10:26.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:10:26.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:10:26.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:10:26.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:10:26.578 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:10:26.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4271cdc-3827-4921-8177-f9d1daa5379e 2024-04-25T09:10:26.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d4271cdc-3827-4921-8177-f9d1daa5379e' 2024-04-25T09:10:26.579 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 d4271cdc-3827-4921-8177-f9d1daa5379e 2024-04-25T09:10:26.579 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:10:26.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACHipmT6VlJBAAX2MKVIQX9JXOwc/Iizv5pg== 2024-04-25T09:10:26.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACHipmT6VlJBAAX2MKVIQX9JXOwc/Iizv5pg=="}' 2024-04-25T09:10:26.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4271cdc-3827-4921-8177-f9d1daa5379e -i td/pg-split-merge/0/new.json 2024-04-25T09:10:26.900 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:10:26.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:10:26.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACHipmT6VlJBAAX2MKVIQX9JXOwc/Iizv5pg== --osd-uuid d4271cdc-3827-4921-8177-f9d1daa5379e 2024-04-25T09:10:26.952 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:26.964+0000 7f3b1049e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:26.969 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:26.980+0000 7f3b1049e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:26.984+0000 7f3b1049e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:26.984+0000 7f3b1049e7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:10:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:26.984+0000 7f3b1049e7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:10:26.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:26.984+0000 7f3b1049e7c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:10:27.009 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:27.020+0000 7f3b1049e7c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:10:33.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:10:33.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:10:33.204 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:10:33.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:10:33.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:10:33.802 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:10:33.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:10:33.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:10:33.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:10:33.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:10:33.804 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:10:33.837 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:33.848+0000 7fb5fd0567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:33.857 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:33.868+0000 7fb5fd0567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:33.858 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:33.872+0000 7fb5fd0567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:34.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:10:34.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:10:34.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:10:34.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:10:34.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:10:34.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:34.212 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:10:34.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:10:34.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:34.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:34.424 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:34.436+0000 7fb5fd0567c0 -1 Falling back to public interface 2024-04-25T09:10:34.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:35.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:35.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:35.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:10:35.647 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:10:35.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:35.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:36.050 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:37.024 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:10:37.036+0000 7fb5fd0567c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:10:37.051 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:37.051 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:10:37.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:37.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:10:37.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:37.052 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:37.740 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:38.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:38.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:38.743 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:10:38.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:10:38.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:38.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:39.194 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:40.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:40.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:40.196 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:10:40.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:10:40.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:40.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:40.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:41.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:41.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:41.623 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:10:41.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:10:41.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:41.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:42.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:43.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:43.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:43.082 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:10:43.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:10:43.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:43.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:43.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:44.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:44.540 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:10:44.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:44.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:10:44.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:44.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:44.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:45.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:45.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:45.998 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:10:45.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:10:45.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:45.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:46.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:47.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:47.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:47.448 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:10:47.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:10:47.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:47.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:47.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:48.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:48.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:48.963 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:10:48.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:10:48.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:48.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:49.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:50.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:50.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:50.373 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:10:50.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:10:50.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:50.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:50.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:51.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:51.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:51.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:10:51.836 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:10:51.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:51.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:52.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:53.462 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:53.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:53.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:10:53.463 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:10:53.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:53.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:54.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:55.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:55.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:55.454 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:10:55.454 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:10:55.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:55.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:56.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:57.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:57.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:57.353 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:10:57.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:10:57.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:57.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:58.256 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:59.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:59.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:59.258 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:10:59.258 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:10:59.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:59.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:59.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:00.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:00.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:00.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:11:00.699 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:11:00.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:00.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:01.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:02.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:02.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:02.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:11:02.300 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:11:02.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:02.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:02.737 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2355146184,v1:127.0.0.1:6803/2355146184] [v2:127.0.0.1:6804/2355146184,v1:127.0.0.1:6805/2355146184] exists,up d4271cdc-3827-4921-8177-f9d1daa5379e 2024-04-25T09:11:02.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:11:02.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:11:02.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:11:02.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:169: TEST_import_after_split: ceph osd pool create foo 1 2024-04-25T09:11:03.291 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'foo' already exists 2024-04-25T09:11:03.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:170: TEST_import_after_split: wait_for_clean 2024-04-25T09:11:03.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:11:03.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:11:03.307 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:11:03.307 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:11:03.308 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:11:03.308 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:11:03.308 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:11:03.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:11:03.309 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:11:03.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:11:03.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:11:03.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:11:03.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:11:03.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:11:03.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:11:03.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:11:03.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:11:03.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:11:03.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:11:04.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836485 2024-04-25T09:11:04.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836485 2024-04-25T09:11:04.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836485' 2024-04-25T09:11:04.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:11:04.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836485 2024-04-25T09:11:04.075 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:11:04.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:11:04.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:11:04.076 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836485 2024-04-25T09:11:04.077 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836485 2024-04-25T09:11:04.077 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836485 2024-04-25T09:11:04.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-04-25T09:11:04.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:11:04.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836483 -lt 21474836485 2024-04-25T09:11:04.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:11:05.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:11:05.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:11:05.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836485 2024-04-25T09:11:05.982 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:11:05.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:11:05.983 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:11:06.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:11:06.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:11:06.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:11:06.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:11:06.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:11:06.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:11:06.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:11:06.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:11:06.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:11:06.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:11:06.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:11:06.973 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:11:07.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:11:07.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:11:07.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:11:07.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:171: TEST_import_after_split: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-04-25T09:11:07.548 INFO:tasks.workunit.client.0.smithi088.stdout:hints = 1 2024-04-25T09:11:07.548 INFO:tasks.workunit.client.0.smithi088.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-04-25T09:11:07.548 INFO:tasks.workunit.client.0.smithi088.stdout:Object prefix: benchmark_data_smithi088_474635 2024-04-25T09:11:07.548 INFO:tasks.workunit.client.0.smithi088.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:11:07.548 INFO:tasks.workunit.client.0.smithi088.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:11:08.548 INFO:tasks.workunit.client.0.smithi088.stdout: 1 16 128 112 0.109362 0.109375 0.117281 0.138808 2024-04-25T09:11:09.548 INFO:tasks.workunit.client.0.smithi088.stdout: 2 16 241 225 0.109849 0.110352 0.133611 0.135127 2024-04-25T09:11:10.548 INFO:tasks.workunit.client.0.smithi088.stdout: 3 16 368 352 0.114567 0.124023 0.117257 0.134081 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Total time run: 3.07139 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Total writes made: 368 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Write size: 1024 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Object size: 1024 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Bandwidth (MB/sec): 0.117007 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Bandwidth: 0.00818994 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Max bandwidth (MB/sec): 0.124023 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Min bandwidth (MB/sec): 0.109375 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Average IOPS: 119 2024-04-25T09:11:11.549 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev IOPS: 8.3865 2024-04-25T09:11:11.550 INFO:tasks.workunit.client.0.smithi088.stdout:Max IOPS: 127 2024-04-25T09:11:11.550 INFO:tasks.workunit.client.0.smithi088.stdout:Min IOPS: 112 2024-04-25T09:11:11.550 INFO:tasks.workunit.client.0.smithi088.stdout:Average Latency(s): 0.133337 2024-04-25T09:11:11.550 INFO:tasks.workunit.client.0.smithi088.stdout:Stddev Latency(s): 0.0301498 2024-04-25T09:11:11.550 INFO:tasks.workunit.client.0.smithi088.stdout:Max latency(s): 0.245134 2024-04-25T09:11:11.550 INFO:tasks.workunit.client.0.smithi088.stdout:Min latency(s): 0.099931 2024-04-25T09:11:11.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:173: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:11:11.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:11:11.624 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:11:11.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:11:11.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:11:11.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:11:11.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:11:11.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:174: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:11:12.210 INFO:tasks.workunit.client.0.smithi088.stderr:Exporting 1.0 info 1.0( v 9'369 (0'0,9'369] local-lis/les=7/8 n=369 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:11:12.224 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0028aea4:::benchmark_data_smithi088_474635_object70:head# 2024-04-25T09:11:12.225 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0072ac6c:::benchmark_data_smithi088_474635_object297:head# 2024-04-25T09:11:12.225 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:00e46afa:::benchmark_data_smithi088_474635_object201:head# 2024-04-25T09:11:12.226 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0115a25b:::benchmark_data_smithi088_474635_object67:head# 2024-04-25T09:11:12.227 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:01c50f00:::benchmark_data_smithi088_474635_object301:head# 2024-04-25T09:11:12.228 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:01d30b1e:::benchmark_data_smithi088_474635_object200:head# 2024-04-25T09:11:12.229 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0207f557:::benchmark_data_smithi088_474635_object278:head# 2024-04-25T09:11:12.230 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:029f4278:::benchmark_data_smithi088_474635_object32:head# 2024-04-25T09:11:12.239 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:02e035ca:::benchmark_data_smithi088_474635_object344:head# 2024-04-25T09:11:12.240 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:02f2bbca:::benchmark_data_smithi088_474635_object36:head# 2024-04-25T09:11:12.241 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:04aec993:::benchmark_data_smithi088_474635_object132:head# 2024-04-25T09:11:12.242 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:04e1c09c:::benchmark_data_smithi088_474635_object325:head# 2024-04-25T09:11:12.243 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:04ec75cb:::benchmark_data_smithi088_474635_object295:head# 2024-04-25T09:11:12.245 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:050c3199:::benchmark_data_smithi088_474635_object61:head# 2024-04-25T09:11:12.245 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:065efd69:::benchmark_data_smithi088_474635_object280:head# 2024-04-25T09:11:12.246 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:08630ca9:::benchmark_data_smithi088_474635_object14:head# 2024-04-25T09:11:12.256 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:086d2cde:::benchmark_data_smithi088_474635_object247:head# 2024-04-25T09:11:12.256 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:09bafaeb:::benchmark_data_smithi088_474635_object241:head# 2024-04-25T09:11:12.257 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:09fc5c5f:::benchmark_data_smithi088_474635_object6:head# 2024-04-25T09:11:12.258 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0a5643a9:::benchmark_data_smithi088_474635_object158:head# 2024-04-25T09:11:12.259 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0a95b5cf:::benchmark_data_smithi088_474635_object260:head# 2024-04-25T09:11:12.261 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0e1b1263:::benchmark_data_smithi088_474635_object296:head# 2024-04-25T09:11:12.261 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:0f28134a:::benchmark_data_smithi088_474635_object98:head# 2024-04-25T09:11:12.262 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:10419da0:::benchmark_data_smithi088_474635_object148:head# 2024-04-25T09:11:12.263 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:10dac8b9:::benchmark_data_smithi088_474635_object63:head# 2024-04-25T09:11:12.263 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:10dca41f:::benchmark_data_smithi088_474635_object244:head# 2024-04-25T09:11:12.264 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1203d321:::benchmark_data_smithi088_474635_object193:head# 2024-04-25T09:11:12.264 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:123ce8ec:::benchmark_data_smithi088_474635_object203:head# 2024-04-25T09:11:12.265 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:12458a43:::benchmark_data_smithi088_474635_object121:head# 2024-04-25T09:11:12.265 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:125210df:::benchmark_data_smithi088_474635_object253:head# 2024-04-25T09:11:12.266 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:12d6f1fe:::benchmark_data_smithi088_474635_object113:head# 2024-04-25T09:11:12.267 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:12f2d4aa:::benchmark_data_smithi088_474635_object235:head# 2024-04-25T09:11:12.267 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:134281fb:::benchmark_data_smithi088_474635_object266:head# 2024-04-25T09:11:12.267 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:13537a3f:::benchmark_data_smithi088_474635_object57:head# 2024-04-25T09:11:12.268 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:13663042:::benchmark_data_smithi088_474635_object137:head# 2024-04-25T09:11:12.268 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:142c38e4:::benchmark_data_smithi088_474635_object367:head# 2024-04-25T09:11:12.268 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:14634413:::benchmark_data_smithi088_474635_object129:head# 2024-04-25T09:11:12.268 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:16f86b08:::benchmark_data_smithi088_474635_object178:head# 2024-04-25T09:11:12.269 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1723e09a:::benchmark_data_smithi088_474635_object147:head# 2024-04-25T09:11:12.269 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:17c6165f:::benchmark_data_smithi088_474635_object232:head# 2024-04-25T09:11:12.270 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:17cea811:::benchmark_data_smithi088_474635_object223:head# 2024-04-25T09:11:12.270 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:180ab51c:::benchmark_data_smithi088_474635_object356:head# 2024-04-25T09:11:12.271 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:18bbb655:::benchmark_data_smithi088_474635_object65:head# 2024-04-25T09:11:12.271 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:195c99ac:::benchmark_data_smithi088_474635_object279:head# 2024-04-25T09:11:12.272 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1b89a1de:::benchmark_data_smithi088_474635_object68:head# 2024-04-25T09:11:12.273 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1cd35e0a:::benchmark_data_smithi088_474635_object261:head# 2024-04-25T09:11:12.273 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1cdffef9:::benchmark_data_smithi088_474635_object119:head# 2024-04-25T09:11:12.274 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1df233e4:::benchmark_data_smithi088_474635_object82:head# 2024-04-25T09:11:12.275 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:1f77643f:::benchmark_data_smithi088_474635_object197:head# 2024-04-25T09:11:12.275 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2105e4c1:::benchmark_data_smithi088_474635_object327:head# 2024-04-25T09:11:12.276 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:21389a19:::benchmark_data_smithi088_474635_object284:head# 2024-04-25T09:11:12.277 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2157c102:::benchmark_data_smithi088_474635_object263:head# 2024-04-25T09:11:12.277 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:216515d2:::benchmark_data_smithi088_474635_object190:head# 2024-04-25T09:11:12.278 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:21a850ef:::benchmark_data_smithi088_474635_object66:head# 2024-04-25T09:11:12.279 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:220dddb9:::benchmark_data_smithi088_474635_object103:head# 2024-04-25T09:11:12.279 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2314d873:::benchmark_data_smithi088_474635_object156:head# 2024-04-25T09:11:12.280 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:237fddad:::benchmark_data_smithi088_474635_object245:head# 2024-04-25T09:11:12.281 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2413ea50:::benchmark_data_smithi088_474635_object303:head# 2024-04-25T09:11:12.281 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2509de11:::benchmark_data_smithi088_474635_object124:head# 2024-04-25T09:11:12.281 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:25be00d9:::benchmark_data_smithi088_474635_object55:head# 2024-04-25T09:11:12.282 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:27662daf:::benchmark_data_smithi088_474635_object171:head# 2024-04-25T09:11:12.283 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:285bfb1b:::benchmark_data_smithi088_474635_object75:head# 2024-04-25T09:11:12.284 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:294df2b5:::benchmark_data_smithi088_474635_object194:head# 2024-04-25T09:11:12.284 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:29c92c62:::benchmark_data_smithi088_474635_object108:head# 2024-04-25T09:11:12.285 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2aa517bd:::benchmark_data_smithi088_474635_object104:head# 2024-04-25T09:11:12.285 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2aca7793:::benchmark_data_smithi088_474635_object160:head# 2024-04-25T09:11:12.286 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2b6ca883:::benchmark_data_smithi088_474635_object96:head# 2024-04-25T09:11:12.287 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2c329146:::benchmark_data_smithi088_474635_object312:head# 2024-04-25T09:11:12.287 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2c676fa6:::benchmark_data_smithi088_474635_object216:head# 2024-04-25T09:11:12.288 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2cb7968f:::benchmark_data_smithi088_474635_object225:head# 2024-04-25T09:11:12.289 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2ce995e4:::benchmark_data_smithi088_474635_object60:head# 2024-04-25T09:11:12.289 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2d705aec:::benchmark_data_smithi088_474635_object283:head# 2024-04-25T09:11:12.290 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2d7096ed:::benchmark_data_smithi088_474635_object362:head# 2024-04-25T09:11:12.291 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2e0a4ce6:::benchmark_data_smithi088_474635_object205:head# 2024-04-25T09:11:12.291 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2e2669dc:::benchmark_data_smithi088_474635_object143:head# 2024-04-25T09:11:12.292 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2e2679ac:::benchmark_data_smithi088_474635_object273:head# 2024-04-25T09:11:12.293 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2e5420b7:::benchmark_data_smithi088_474635_object272:head# 2024-04-25T09:11:12.293 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f121d11:::benchmark_data_smithi088_474635_object179:head# 2024-04-25T09:11:12.294 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f2ca407:::benchmark_data_smithi088_474635_object149:head# 2024-04-25T09:11:12.295 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f2eaa6f:::benchmark_data_smithi088_474635_object328:head# 2024-04-25T09:11:12.295 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f644dd2:::benchmark_data_smithi088_474635_object191:head# 2024-04-25T09:11:12.296 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:2f6a8b03:::benchmark_data_smithi088_474635_object360:head# 2024-04-25T09:11:12.297 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:319d1b1b:::benchmark_data_smithi088_474635_object281:head# 2024-04-25T09:11:12.297 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:32caca73:::benchmark_data_smithi088_474635_object308:head# 2024-04-25T09:11:12.298 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:34829fd5:::benchmark_data_smithi088_474635_object248:head# 2024-04-25T09:11:12.299 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:34fe1da9:::benchmark_data_smithi088_474635_object286:head# 2024-04-25T09:11:12.299 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:35e0e924:::benchmark_data_smithi088_474635_object198:head# 2024-04-25T09:11:12.300 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:369feb74:::benchmark_data_smithi088_474635_object111:head# 2024-04-25T09:11:12.301 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:36d320eb:::benchmark_data_smithi088_474635_object355:head# 2024-04-25T09:11:12.301 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3712cfef:::benchmark_data_smithi088_474635_object230:head# 2024-04-25T09:11:12.302 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:38365cc5:::benchmark_data_smithi088_474635_object242:head# 2024-04-25T09:11:12.303 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:38478e6e:::benchmark_data_smithi088_474635_object109:head# 2024-04-25T09:11:12.303 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:38fa84e3:::benchmark_data_smithi088_474635_object320:head# 2024-04-25T09:11:12.304 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:396a89f4:::benchmark_data_smithi088_474635_object45:head# 2024-04-25T09:11:12.305 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:399f132d:::benchmark_data_smithi088_474635_object307:head# 2024-04-25T09:11:12.305 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3a9610aa:::benchmark_data_smithi088_474635_object83:head# 2024-04-25T09:11:12.306 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3ba25773:::benchmark_data_smithi088_474635_object114:head# 2024-04-25T09:11:12.307 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3c5f6b6b:::benchmark_data_smithi088_474635_object2:head# 2024-04-25T09:11:12.307 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3d19860c:::benchmark_data_smithi088_474635_object251:head# 2024-04-25T09:11:12.308 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3d540a92:::benchmark_data_smithi088_474635_object229:head# 2024-04-25T09:11:12.309 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3e3230ca:::benchmark_data_smithi088_474635_object212:head# 2024-04-25T09:11:12.309 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:3febb60b:::benchmark_data_smithi088_474635_object50:head# 2024-04-25T09:11:12.310 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4056bc59:::benchmark_data_smithi088_474635_object240:head# 2024-04-25T09:11:12.310 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:40dc14fa:::benchmark_data_smithi088_474635_object180:head# 2024-04-25T09:11:12.311 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:410cb459:::benchmark_data_smithi088_474635_object151:head# 2024-04-25T09:11:12.312 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:418971ee:::benchmark_data_smithi088_474635_object174:head# 2024-04-25T09:11:12.313 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:428a380e:::benchmark_data_smithi088_474635_object293:head# 2024-04-25T09:11:12.313 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:428b1a67:::benchmark_data_smithi088_474635_object353:head# 2024-04-25T09:11:12.314 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:42922198:::benchmark_data_smithi088_474635_object145:head# 2024-04-25T09:11:12.315 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:42be703d:::benchmark_data_smithi088_474635_object165:head# 2024-04-25T09:11:12.315 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:465bb8d4:::benchmark_data_smithi088_474635_object365:head# 2024-04-25T09:11:12.316 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:47bb0a3c:::benchmark_data_smithi088_474635_object97:head# 2024-04-25T09:11:12.316 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:47f18852:::benchmark_data_smithi088_474635_object346:head# 2024-04-25T09:11:12.317 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4897fda7:::benchmark_data_smithi088_474635_object28:head# 2024-04-25T09:11:12.318 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:48f1a6ac:::benchmark_data_smithi088_474635_object154:head# 2024-04-25T09:11:12.318 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4a21aef9:::benchmark_data_smithi088_474635_object93:head# 2024-04-25T09:11:12.319 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4aa94f54:::benchmark_data_smithi088_474635_object224:head# 2024-04-25T09:11:12.320 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4bbd9dc9:::benchmark_data_smithi088_474635_object88:head# 2024-04-25T09:11:12.320 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4c404371:::benchmark_data_smithi088_474635_object184:head# 2024-04-25T09:11:12.321 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4c6221a7:::benchmark_data_smithi088_474635_object126:head# 2024-04-25T09:11:12.321 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4f664fa4:::benchmark_data_smithi088_474635_object220:head# 2024-04-25T09:11:12.322 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4ff71927:::benchmark_data_smithi088_474635_object288:head# 2024-04-25T09:11:12.323 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:4ff83a37:::benchmark_data_smithi088_474635_object262:head# 2024-04-25T09:11:12.323 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:503dfc05:::benchmark_data_smithi088_474635_object23:head# 2024-04-25T09:11:12.324 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:506ac00a:::benchmark_data_smithi088_474635_object141:head# 2024-04-25T09:11:12.325 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:513ae8d4:::benchmark_data_smithi088_474635_object258:head# 2024-04-25T09:11:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:51b7b0b5:::benchmark_data_smithi088_474635_object290:head# 2024-04-25T09:11:12.326 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:52dad3af:::benchmark_data_smithi088_474635_object231:head# 2024-04-25T09:11:12.327 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:52fda022:::benchmark_data_smithi088_474635_object311:head# 2024-04-25T09:11:12.328 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:55034a6f:::benchmark_data_smithi088_474635_object316:head# 2024-04-25T09:11:12.328 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:56076090:::benchmark_data_smithi088_474635_object133:head# 2024-04-25T09:11:12.329 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:56538934:::benchmark_data_smithi088_474635_object5:head# 2024-04-25T09:11:12.330 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5721e5f8:::benchmark_data_smithi088_474635_object74:head# 2024-04-25T09:11:12.330 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5745116f:::benchmark_data_smithi088_474635_object233:head# 2024-04-25T09:11:12.331 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:579659b2:::benchmark_data_smithi088_474635_object215:head# 2024-04-25T09:11:12.332 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:57cb171b:::benchmark_data_smithi088_474635_object252:head# 2024-04-25T09:11:12.332 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5837f004:::benchmark_data_smithi088_474635_object208:head# 2024-04-25T09:11:12.333 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:59b91b69:::benchmark_data_smithi088_474635_object357:head# 2024-04-25T09:11:12.334 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5beb7e41:::benchmark_data_smithi088_474635_object72:head# 2024-04-25T09:11:12.334 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5d37aaa5:::benchmark_data_smithi088_474635_object289:head# 2024-04-25T09:11:12.335 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5d467473:::benchmark_data_smithi088_474635_object304:head# 2024-04-25T09:11:12.335 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5e2f8898:::benchmark_data_smithi088_474635_object213:head# 2024-04-25T09:11:12.336 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:5fda8027:::benchmark_data_smithi088_474635_object256:head# 2024-04-25T09:11:12.337 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:61ed8bbc:::benchmark_data_smithi088_474635_object116:head# 2024-04-25T09:11:12.337 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:620fc497:::benchmark_data_smithi088_474635_object257:head# 2024-04-25T09:11:12.338 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:62363145:::benchmark_data_smithi088_474635_object125:head# 2024-04-25T09:11:12.338 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:625044ca:::benchmark_data_smithi088_474635_object29:head# 2024-04-25T09:11:12.340 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:637d890b:::benchmark_data_smithi088_474635_object285:head# 2024-04-25T09:11:12.340 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:63bee89e:::benchmark_data_smithi088_474635_object195:head# 2024-04-25T09:11:12.341 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:673a8823:::benchmark_data_smithi088_474635_object71:head# 2024-04-25T09:11:12.342 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:67c9fe8c:::benchmark_data_smithi088_474635_object313:head# 2024-04-25T09:11:12.342 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:686141c3:::benchmark_data_smithi088_474635_object352:head# 2024-04-25T09:11:12.343 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:697aefc7:::benchmark_data_smithi088_474635_object142:head# 2024-04-25T09:11:12.344 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6a2142c4:::benchmark_data_smithi088_474635_object222:head# 2024-04-25T09:11:12.345 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6a582a71:::benchmark_data_smithi088_474635_object21:head# 2024-04-25T09:11:12.345 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6ac0d24f:::benchmark_data_smithi088_474635_object211:head# 2024-04-25T09:11:12.346 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6b27391b:::benchmark_data_smithi088_474635_object90:head# 2024-04-25T09:11:12.346 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6b2f1bd4:::benchmark_data_smithi088_474635_object16:head# 2024-04-25T09:11:12.347 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6bdc7245:::benchmark_data_smithi088_474635_object89:head# 2024-04-25T09:11:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6c85a23d:::benchmark_data_smithi088_474635_object130:head# 2024-04-25T09:11:12.348 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6c97d0c1:::benchmark_data_smithi088_474635_object44:head# 2024-04-25T09:11:12.349 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6dd55281:::benchmark_data_smithi088_474635_object333:head# 2024-04-25T09:11:12.350 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6eaeb124:::benchmark_data_smithi088_474635_object3:head# 2024-04-25T09:11:12.351 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:6ecb117e:::benchmark_data_smithi088_474635_object182:head# 2024-04-25T09:11:12.351 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7029915e:::benchmark_data_smithi088_474635_object217:head# 2024-04-25T09:11:12.352 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:70425cf6:::benchmark_data_smithi088_474635_object35:head# 2024-04-25T09:11:12.353 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:70874157:::benchmark_data_smithi088_474635_object170:head# 2024-04-25T09:11:12.353 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:725e36bc:::benchmark_data_smithi088_474635_object269:head# 2024-04-25T09:11:12.354 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:72b36e3d:::benchmark_data_smithi088_474635_object153:head# 2024-04-25T09:11:12.355 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:738ae35b:::benchmark_data_smithi088_474635_object122:head# 2024-04-25T09:11:12.355 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:74adfe87:::benchmark_data_smithi088_474635_object302:head# 2024-04-25T09:11:12.356 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:758c96f1:::benchmark_data_smithi088_474635_object214:head# 2024-04-25T09:11:12.357 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:75e520f7:::benchmark_data_smithi088_474635_object169:head# 2024-04-25T09:11:12.357 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:76faa396:::benchmark_data_smithi088_474635_object128:head# 2024-04-25T09:11:12.358 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:782b27ad:::benchmark_data_smithi088_474635_object350:head# 2024-04-25T09:11:12.359 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7a97a86c:::benchmark_data_smithi088_474635_object26:head# 2024-04-25T09:11:12.359 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7abe2685:::benchmark_data_smithi088_474635_object299:head# 2024-04-25T09:11:12.360 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7bc6a2b5:::benchmark_data_smithi088_474635_object347:head# 2024-04-25T09:11:12.361 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7c492c96:::benchmark_data_smithi088_474635_object228:head# 2024-04-25T09:11:12.361 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7ce6829c:::benchmark_data_smithi088_474635_object0:head# 2024-04-25T09:11:12.362 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7d7fe9bd:::benchmark_data_smithi088_474635_object106:head# 2024-04-25T09:11:12.362 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7e52051d:::benchmark_data_smithi088_474635_object24:head# 2024-04-25T09:11:12.363 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7f01f963:::benchmark_data_smithi088_474635_object25:head# 2024-04-25T09:11:12.363 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7f887662:::benchmark_data_smithi088_474635_object64:head# 2024-04-25T09:11:12.364 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:7f8a93a8:::benchmark_data_smithi088_474635_object139:head# 2024-04-25T09:11:12.365 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8072f058:::benchmark_data_smithi088_474635_object172:head# 2024-04-25T09:11:12.365 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:80eaa411:::benchmark_data_smithi088_474635_object177:head# 2024-04-25T09:11:12.366 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8154841a:::benchmark_data_smithi088_474635_object227:head# 2024-04-25T09:11:12.366 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8154f227:::benchmark_data_smithi088_474635_object276:head# 2024-04-25T09:11:12.366 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8191b4d4:::benchmark_data_smithi088_474635_object332:head# 2024-04-25T09:11:12.367 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:82d242f9:::benchmark_data_smithi088_474635_object62:head# 2024-04-25T09:11:12.368 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:82fd563c:::benchmark_data_smithi088_474635_object183:head# 2024-04-25T09:11:12.368 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8308ab9e:::benchmark_data_smithi088_474635_object41:head# 2024-04-25T09:11:12.369 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8559d1df:::benchmark_data_smithi088_474635_object112:head# 2024-04-25T09:11:12.370 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:857941d3:::benchmark_data_smithi088_474635_object168:head# 2024-04-25T09:11:12.370 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:85d243ce:::benchmark_data_smithi088_474635_object42:head# 2024-04-25T09:11:12.371 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8662071c:::benchmark_data_smithi088_474635_object39:head# 2024-04-25T09:11:12.371 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8663fd45:::benchmark_data_smithi088_474635_object4:head# 2024-04-25T09:11:12.372 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:86798526:::benchmark_data_smithi088_474635_object300:head# 2024-04-25T09:11:12.372 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:86958939:::benchmark_data_smithi088_474635_object255:head# 2024-04-25T09:11:12.373 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:86d5de05:::benchmark_data_smithi088_474635_object53:head# 2024-04-25T09:11:12.374 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:87295d2d:::benchmark_data_smithi088_474635_object340:head# 2024-04-25T09:11:12.374 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:895bbe02:::benchmark_data_smithi088_474635_object204:head# 2024-04-25T09:11:12.375 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8c544bab:::benchmark_data_smithi088_474635_object187:head# 2024-04-25T09:11:12.376 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8dc506d9:::benchmark_data_smithi088_474635_object309:head# 2024-04-25T09:11:12.376 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8e17a3ba:::benchmark_data_smithi088_474635_object30:head# 2024-04-25T09:11:12.377 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8e193346:::benchmark_data_smithi088_474635_object249:head# 2024-04-25T09:11:12.378 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8e813d25:::benchmark_data_smithi088_474635_object162:head# 2024-04-25T09:11:12.378 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8f067153:::benchmark_data_smithi088_474635_object236:head# 2024-04-25T09:11:12.379 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8f50ce47:::benchmark_data_smithi088_474635_object48:head# 2024-04-25T09:11:12.379 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:8f535138:::benchmark_data_smithi088_474635_object274:head# 2024-04-25T09:11:12.380 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:91e6cc24:::benchmark_data_smithi088_474635_object123:head# 2024-04-25T09:11:12.381 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9201d91f:::benchmark_data_smithi088_474635_object10:head# 2024-04-25T09:11:12.381 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:92193071:::benchmark_data_smithi088_474635_object173:head# 2024-04-25T09:11:12.382 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:923c5ad1:::benchmark_data_smithi088_474635_object254:head# 2024-04-25T09:11:12.382 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:92759122:::benchmark_data_smithi088_474635_object206:head# 2024-04-25T09:11:12.383 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9312eb3e:::benchmark_data_smithi088_474635_object131:head# 2024-04-25T09:11:12.384 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9357f6c0:::benchmark_data_smithi088_474635_object239:head# 2024-04-25T09:11:12.385 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:937bf92d:::benchmark_data_smithi088_474635_object334:head# 2024-04-25T09:11:12.385 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9439d424:::benchmark_data_smithi088_474635_object270:head# 2024-04-25T09:11:12.386 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:957c4957:::benchmark_data_smithi088_474635_object292:head# 2024-04-25T09:11:12.386 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9605c18c:::benchmark_data_smithi088_474635_object84:head# 2024-04-25T09:11:12.387 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:97dc4359:::benchmark_data_smithi088_474635_object91:head# 2024-04-25T09:11:12.388 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:989d0658:::benchmark_data_smithi088_474635_object47:head# 2024-04-25T09:11:12.388 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:99596455:::benchmark_data_smithi088_474635_object18:head# 2024-04-25T09:11:12.389 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9a5c4743:::benchmark_data_smithi088_474635_object43:head# 2024-04-25T09:11:12.389 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9a719784:::benchmark_data_smithi088_474635_object12:head# 2024-04-25T09:11:12.390 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9acc2858:::benchmark_data_smithi088_474635_object338:head# 2024-04-25T09:11:12.390 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9ae892d9:::benchmark_data_smithi088_474635_object335:head# 2024-04-25T09:11:12.391 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9c214df4:::benchmark_data_smithi088_474635_object339:head# 2024-04-25T09:11:12.391 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9cf0af18:::benchmark_data_smithi088_474635_object146:head# 2024-04-25T09:11:12.392 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:9d679c58:::benchmark_data_smithi088_474635_object155:head# 2024-04-25T09:11:12.393 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a154514b:::benchmark_data_smithi088_474635_object243:head# 2024-04-25T09:11:12.393 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a1c6e016:::benchmark_data_smithi088_474635_object199:head# 2024-04-25T09:11:12.394 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a22943b6:::benchmark_data_smithi088_474635_object287:head# 2024-04-25T09:11:12.394 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a2867657:::benchmark_data_smithi088_474635_object87:head# 2024-04-25T09:11:12.395 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a2dbd594:::benchmark_data_smithi088_474635_object310:head# 2024-04-25T09:11:12.396 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a3d32bff:::benchmark_data_smithi088_474635_object101:head# 2024-04-25T09:11:12.396 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a43a8dab:::benchmark_data_smithi088_474635_object86:head# 2024-04-25T09:11:12.397 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a5753aec:::benchmark_data_smithi088_474635_object31:head# 2024-04-25T09:11:12.398 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a6b94731:::benchmark_data_smithi088_474635_object161:head# 2024-04-25T09:11:12.398 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a74259fa:::benchmark_data_smithi088_474635_object166:head# 2024-04-25T09:11:12.399 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a916d9c4:::benchmark_data_smithi088_474635_object341:head# 2024-04-25T09:11:12.399 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a92c6ab4:::benchmark_data_smithi088_474635_object102:head# 2024-04-25T09:11:12.400 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:a9ada565:::benchmark_data_smithi088_474635_object134:head# 2024-04-25T09:11:12.401 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:aa9ac8f2:::benchmark_data_smithi088_474635_object219:head# 2024-04-25T09:11:12.402 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:aac5607b:::benchmark_data_smithi088_474635_object321:head# 2024-04-25T09:11:12.402 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ab2bd6b0:::benchmark_data_smithi088_474635_object361:head# 2024-04-25T09:11:12.403 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ab5f28f6:::benchmark_data_smithi088_474635_object34:head# 2024-04-25T09:11:12.404 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ab8ca43f:::benchmark_data_smithi088_474635_object326:head# 2024-04-25T09:11:12.405 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:acbd0c95:::benchmark_data_smithi088_474635_object138:head# 2024-04-25T09:11:12.406 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ad03517a:::benchmark_data_smithi088_474635_object267:head# 2024-04-25T09:11:12.406 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ad8f42e7:::benchmark_data_smithi088_474635_object330:head# 2024-04-25T09:11:12.407 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b3161fff:::benchmark_data_smithi088_474635_object94:head# 2024-04-25T09:11:12.408 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b34353cf:::benchmark_data_smithi088_474635_object185:head# 2024-04-25T09:11:12.408 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b3d5afaf:::benchmark_data_smithi088_474635_object192:head# 2024-04-25T09:11:12.409 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b4a70d6d:::benchmark_data_smithi088_474635_object136:head# 2024-04-25T09:11:12.409 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b4aa86e1:::benchmark_data_smithi088_474635_object115:head# 2024-04-25T09:11:12.410 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b4efc706:::benchmark_data_smithi088_474635_object150:head# 2024-04-25T09:11:12.410 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b6a56dd3:::benchmark_data_smithi088_474635_object221:head# 2024-04-25T09:11:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b6b27c77:::benchmark_data_smithi088_474635_object364:head# 2024-04-25T09:11:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b6d0fb40:::benchmark_data_smithi088_474635_object366:head# 2024-04-25T09:11:12.411 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b8518f13:::benchmark_data_smithi088_474635_object17:head# 2024-04-25T09:11:12.413 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:11:12.413 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ba19a0a4:::benchmark_data_smithi088_474635_object343:head# 2024-04-25T09:11:12.414 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bacf247d:::benchmark_data_smithi088_474635_object342:head# 2024-04-25T09:11:12.414 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bd320262:::benchmark_data_smithi088_474635_object349:head# 2024-04-25T09:11:12.415 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bd48bd40:::benchmark_data_smithi088_474635_object22:head# 2024-04-25T09:11:12.416 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bd5991cf:::benchmark_data_smithi088_474635_object33:head# 2024-04-25T09:11:12.416 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bec41a8c:::benchmark_data_smithi088_474635_object85:head# 2024-04-25T09:11:12.417 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:bf6a0f1a:::benchmark_data_smithi088_474635_object13:head# 2024-04-25T09:11:12.417 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c0e5028f:::benchmark_data_smithi088_474635_object324:head# 2024-04-25T09:11:12.418 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c11e469d:::benchmark_data_smithi088_474635_object77:head# 2024-04-25T09:11:12.418 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c136c80d:::benchmark_data_smithi088_474635_object207:head# 2024-04-25T09:11:12.419 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c26ea520:::benchmark_data_smithi088_474635_object92:head# 2024-04-25T09:11:12.420 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c40ced02:::benchmark_data_smithi088_474635_object277:head# 2024-04-25T09:11:12.420 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c47f6c72:::benchmark_data_smithi088_474635_object305:head# 2024-04-25T09:11:12.421 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c617de94:::benchmark_data_smithi088_474635_object291:head# 2024-04-25T09:11:12.421 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c6181fd8:::benchmark_data_smithi088_474635_object105:head# 2024-04-25T09:11:12.422 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c6992680:::benchmark_data_smithi088_474635_object268:head# 2024-04-25T09:11:12.422 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c714c83e:::benchmark_data_smithi088_474635_object202:head# 2024-04-25T09:11:12.423 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c7263d37:::benchmark_data_smithi088_474635_object189:head# 2024-04-25T09:11:12.424 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c7c7f9c1:::benchmark_data_smithi088_474635_object294:head# 2024-04-25T09:11:12.424 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c80fc382:::benchmark_data_smithi088_474635_object40:head# 2024-04-25T09:11:12.425 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c98d5fcc:::benchmark_data_smithi088_474635_object19:head# 2024-04-25T09:11:12.426 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:c9e15eb4:::benchmark_data_smithi088_474635_object9:head# 2024-04-25T09:11:12.426 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ca7c35cc:::benchmark_data_smithi088_474635_object73:head# 2024-04-25T09:11:12.427 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ca86eae0:::benchmark_data_smithi088_474635_object100:head# 2024-04-25T09:11:12.427 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:caaa7e6f:::benchmark_data_smithi088_474635_object348:head# 2024-04-25T09:11:12.428 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cbb7c498:::benchmark_data_smithi088_474635_object20:head# 2024-04-25T09:11:12.428 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cbfcab2c:::benchmark_data_smithi088_474635_object117:head# 2024-04-25T09:11:12.429 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ccb4b73f:::benchmark_data_smithi088_474635_object59:head# 2024-04-25T09:11:12.430 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cd545eae:::benchmark_data_smithi088_474635_object157:head# 2024-04-25T09:11:12.430 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cd692c8e:::benchmark_data_smithi088_474635_object81:head# 2024-04-25T09:11:12.431 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:cdf279d0:::benchmark_data_smithi088_474635_object358:head# 2024-04-25T09:11:12.431 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ce4241dd:::benchmark_data_smithi088_474635_object314:head# 2024-04-25T09:11:12.432 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ce7922f5:::benchmark_data_smithi088_474635_object345:head# 2024-04-25T09:11:12.433 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d06c083a:::benchmark_data_smithi088_474635_object275:head# 2024-04-25T09:11:12.433 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d233b28d:::benchmark_data_smithi088_474635_object317:head# 2024-04-25T09:11:12.434 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d31f9487:::benchmark_data_smithi088_474635_object56:head# 2024-04-25T09:11:12.435 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d3269e90:::benchmark_data_smithi088_474635_object140:head# 2024-04-25T09:11:12.436 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d398c90e:::benchmark_data_smithi088_474635_object237:head# 2024-04-25T09:11:12.437 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d40e39ec:::benchmark_data_smithi088_474635_object95:head# 2024-04-25T09:11:12.437 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d462d81d:::benchmark_data_smithi088_474635_object135:head# 2024-04-25T09:11:12.438 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d4ee28e3:::benchmark_data_smithi088_474635_object337:head# 2024-04-25T09:11:12.439 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d572577d:::benchmark_data_smithi088_474635_object164:head# 2024-04-25T09:11:12.440 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d5c8e00e:::benchmark_data_smithi088_474635_object336:head# 2024-04-25T09:11:12.440 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d5d3de0c:::benchmark_data_smithi088_474635_object38:head# 2024-04-25T09:11:12.441 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d7234313:::benchmark_data_smithi088_474635_object234:head# 2024-04-25T09:11:12.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d741ddf1:::benchmark_data_smithi088_474635_object37:head# 2024-04-25T09:11:12.442 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:d8774faa:::benchmark_data_smithi088_474635_object306:head# 2024-04-25T09:11:12.443 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dad3e05a:::benchmark_data_smithi088_474635_object318:head# 2024-04-25T09:11:12.444 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:db51ceaf:::benchmark_data_smithi088_474635_object167:head# 2024-04-25T09:11:12.444 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dbcddc20:::benchmark_data_smithi088_474635_object144:head# 2024-04-25T09:11:12.445 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dc239daa:::benchmark_data_smithi088_474635_object76:head# 2024-04-25T09:11:12.446 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dc4dbfb6:::benchmark_data_smithi088_474635_object120:head# 2024-04-25T09:11:12.446 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dcbc5866:::benchmark_data_smithi088_474635_object110:head# 2024-04-25T09:11:12.447 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dcf7bf9c:::benchmark_data_smithi088_474635_object186:head# 2024-04-25T09:11:12.448 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dd72c389:::benchmark_data_smithi088_474635_object79:head# 2024-04-25T09:11:12.449 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dd7f00f3:::benchmark_data_smithi088_474635_object322:head# 2024-04-25T09:11:12.450 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ddbf17cd:::benchmark_data_smithi088_474635_object58:head# 2024-04-25T09:11:12.450 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:de74aab6:::benchmark_data_smithi088_474635_object107:head# 2024-04-25T09:11:12.451 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dfcc03c2:::benchmark_data_smithi088_474635_object127:head# 2024-04-25T09:11:12.451 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:dfe8a4f8:::benchmark_data_smithi088_474635_object265:head# 2024-04-25T09:11:12.452 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e29d67fb:::benchmark_data_smithi088_474635_object69:head# 2024-04-25T09:11:12.453 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e2abbb39:::benchmark_data_smithi088_474635_object298:head# 2024-04-25T09:11:12.453 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e3557d2e:::benchmark_data_smithi088_474635_object8:head# 2024-04-25T09:11:12.454 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e52260c6:::benchmark_data_smithi088_474635_object259:head# 2024-04-25T09:11:12.454 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e691a3f1:::benchmark_data_smithi088_474635_object238:head# 2024-04-25T09:11:12.455 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e7e60cd1:::benchmark_data_smithi088_474635_object218:head# 2024-04-25T09:11:12.456 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:e92137d0:::benchmark_data_smithi088_474635_object46:head# 2024-04-25T09:11:12.456 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ea1fc82e:::benchmark_data_smithi088_474635_object271:head# 2024-04-25T09:11:12.457 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ea235d8b:::benchmark_data_smithi088_474635_object329:head# 2024-04-25T09:11:12.458 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ea5df4a2:::benchmark_data_smithi088_474635_object354:head# 2024-04-25T09:11:12.458 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:eae3ed2d:::benchmark_data_smithi088_474635_object196:head# 2024-04-25T09:11:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ebf9a4c3:::benchmark_data_smithi088_474635_object319:head# 2024-04-25T09:11:12.459 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ed8f3a1b:::benchmark_data_smithi088_474635_object80:head# 2024-04-25T09:11:12.460 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ee957fda:::benchmark_data_smithi088_474635_object51:head# 2024-04-25T09:11:12.460 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ef265467:::benchmark_data_smithi088_474635_object15:head# 2024-04-25T09:11:12.461 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f0514099:::benchmark_data_smithi088_474635_object11:head# 2024-04-25T09:11:12.461 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f06dd765:::benchmark_data_smithi088_474635_object282:head# 2024-04-25T09:11:12.462 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f0987f6b:::benchmark_data_smithi088_474635_object264:head# 2024-04-25T09:11:12.462 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f0cdbd0b:::benchmark_data_smithi088_474635_object176:head# 2024-04-25T09:11:12.463 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f225e4b0:::benchmark_data_smithi088_474635_object118:head# 2024-04-25T09:11:12.464 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f2563bb4:::benchmark_data_smithi088_474635_object226:head# 2024-04-25T09:11:12.464 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f2aaec93:::benchmark_data_smithi088_474635_object152:head# 2024-04-25T09:11:12.465 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f2d5ca00:::benchmark_data_smithi088_474635_object159:head# 2024-04-25T09:11:12.466 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f3cd498d:::benchmark_data_smithi088_474635_object78:head# 2024-04-25T09:11:12.466 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f3f30a65:::benchmark_data_smithi088_474635_object163:head# 2024-04-25T09:11:12.467 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f4caeb31:::benchmark_data_smithi088_474635_object315:head# 2024-04-25T09:11:12.468 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f5425fd8:::benchmark_data_smithi088_474635_object27:head# 2024-04-25T09:11:12.469 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f560643f:::benchmark_data_smithi088_474635_object363:head# 2024-04-25T09:11:12.470 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f7085f7a:::benchmark_data_smithi088_474635_object181:head# 2024-04-25T09:11:12.470 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f72b2424:::benchmark_data_smithi088_474635_object331:head# 2024-04-25T09:11:12.471 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f76b0004:::benchmark_data_smithi088_474635_object188:head# 2024-04-25T09:11:12.472 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f82f89f0:::benchmark_data_smithi088_474635_object1:head# 2024-04-25T09:11:12.472 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f8b0d0ea:::benchmark_data_smithi088_474635_object99:head# 2024-04-25T09:11:12.473 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f8c1439b:::benchmark_data_smithi088_474635_object52:head# 2024-04-25T09:11:12.473 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:f8ca900e:::benchmark_data_smithi088_474635_object246:head# 2024-04-25T09:11:12.474 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fc5127b1:::benchmark_data_smithi088_474635_object210:head# 2024-04-25T09:11:12.475 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fc806264:::benchmark_data_smithi088_474635_object359:head# 2024-04-25T09:11:12.475 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fcf4dc66:::benchmark_data_smithi088_474635_object209:head# 2024-04-25T09:11:12.476 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fe726c16:::benchmark_data_smithi088_474635_object49:head# 2024-04-25T09:11:12.477 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:fe7aee35:::benchmark_data_smithi088_474635_object54:head# 2024-04-25T09:11:12.477 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ff268577:::benchmark_data_smithi088_474635_object7:head# 2024-04-25T09:11:12.478 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ff457b2a:::benchmark_data_smithi088_474635_object323:head# 2024-04-25T09:11:12.478 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ff7af289:::benchmark_data_smithi088_474635_object250:head# 2024-04-25T09:11:12.479 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ffa0454c:::benchmark_data_smithi088_474635_object175:head# 2024-04-25T09:11:12.480 INFO:tasks.workunit.client.0.smithi088.stderr:Read #1:ffc40287:::benchmark_data_smithi088_474635_object351:head# 2024-04-25T09:11:12.481 INFO:tasks.workunit.client.0.smithi088.stderr:Export successful 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:175: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:11:12.772 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:11:12.773 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:11:12.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:11:12.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:11:12.776 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:11:12.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:11:12.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:11:12.776 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:11:12.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:11:12.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:11:12.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:11:12.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:11:12.812 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:12.823+0000 7fb1562097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:12.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:12.827+0000 7fb1562097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:12.816 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:12.827+0000 7fb1562097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:13.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:11:13.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:11:13.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:11:13.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:11:13.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:11:13.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:13.212 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:11:13.212 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:11:13.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:13.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:13.383 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:13.395+0000 7fb1562097c0 -1 Falling back to public interface 2024-04-25T09:11:13.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:14.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:14.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:14.687 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:11:14.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:11:14.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:14.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:15.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:16.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:16.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:16.120 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:11:16.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:11:16.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:16.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:16.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:16.647 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:16.659+0000 7fb1562097c0 -1 osd.0 9 log_to_monitors true 2024-04-25T09:11:17.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:17.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:17.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:11:17.539 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:11:17.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:17.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:17.972 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/3016508067,v1:127.0.0.1:6803/3016508067] [v2:127.0.0.1:6804/3016508067,v1:127.0.0.1:6805/3016508067] exists,up d4271cdc-3827-4921-8177-f9d1daa5379e 2024-04-25T09:11:17.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:11:17.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:11:17.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:11:17.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:177: TEST_import_after_split: ceph osd pool set foo pg_num 2 2024-04-25T09:11:18.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:178: TEST_import_after_split: sleep 5 2024-04-25T09:11:23.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: ceph daemon osd.0 perf dump 2024-04-25T09:11:23.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: jq .osd.numpg 2024-04-25T09:11:23.662 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: grep 1 2024-04-25T09:11:23.797 INFO:tasks.workunit.client.0.smithi088.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-25T09:11:23.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:180: TEST_import_after_split: wait_for_clean 2024-04-25T09:11:23.799 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:11:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:11:23.800 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:11:23.800 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:11:23.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:11:23.801 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:11:23.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:11:23.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:11:23.801 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:11:23.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:11:23.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:11:23.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:11:23.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:11:23.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:11:23.907 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:11:24.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:11:24.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:11:24.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:11:24.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:11:24.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607556 2024-04-25T09:11:24.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607556 2024-04-25T09:11:24.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-51539607556' 2024-04-25T09:11:24.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:11:24.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-51539607556 2024-04-25T09:11:24.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:11:24.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:11:24.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-51539607556 2024-04-25T09:11:24.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:11:24.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607556 2024-04-25T09:11:24.511 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 51539607556 2024-04-25T09:11:24.511 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-04-25T09:11:24.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:11:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607554 -lt 51539607556 2024-04-25T09:11:24.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:11:25.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:11:25.930 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:11:26.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607556 -lt 51539607556 2024-04-25T09:11:26.330 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:11:26.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:11:26.331 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:11:26.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:11:26.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:11:26.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:11:26.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:11:26.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:11:26.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:11:26.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:11:26.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:11:27.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:11:27.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:11:27.298 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:11:27.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:11:27.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:182: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:11:27.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:11:27.855 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:11:27.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:11:27.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:11:27.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:11:28.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:11:28.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:184: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:11:29.410 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T09:11:29.410 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T09:11:29.411 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:11:29.836 INFO:tasks.workunit.client.0.smithi088.stdout:Remove successful 2024-04-25T09:11:30.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:187: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:11:31.293 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-04-25T09:11:31.293 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.0 2024-04-25T09:11:31.294 INFO:tasks.workunit.client.0.smithi088.stdout:pool 1 pg_num 1 -> 2 2024-04-25T09:11:31.294 INFO:tasks.workunit.client.0.smithi088.stderr: children are 1.1 2024-04-25T09:11:31.294 INFO:tasks.workunit.client.0.smithi088.stderr:pgid 1.0 splits to 1.1 and 1.1 exists 2024-04-25T09:11:31.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:189: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-04-25T09:11:32.235 INFO:tasks.workunit.client.0.smithi088.stdout: marking collection for removal 2024-04-25T09:11:32.235 INFO:tasks.workunit.client.0.smithi088.stdout:setting '_remove' omap key 2024-04-25T09:11:32.235 INFO:tasks.workunit.client.0.smithi088.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:11:32.558 INFO:tasks.workunit.client.0.smithi088.stdout:Remove successful 2024-04-25T09:11:33.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:191: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:11:33.890 INFO:tasks.workunit.client.0.smithi088.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-04-25T09:11:33.890 INFO:tasks.workunit.client.0.smithi088.stdout:Importing pgid 1.0 2024-04-25T09:11:33.891 INFO:tasks.workunit.client.0.smithi088.stdout:pool 1 pg_num 1 -> 2 2024-04-25T09:11:33.891 INFO:tasks.workunit.client.0.smithi088.stderr: children are 1.1 2024-04-25T09:11:33.891 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0028aea4:::benchmark_data_smithi088_474635_object70:head# 2024-04-25T09:11:33.891 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:33.953 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0072ac6c:::benchmark_data_smithi088_474635_object297:head# 2024-04-25T09:11:33.953 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:33.987 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:00e46afa:::benchmark_data_smithi088_474635_object201:head# 2024-04-25T09:11:33.987 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.020 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0115a25b:::benchmark_data_smithi088_474635_object67:head# 2024-04-25T09:11:34.020 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.053 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:01c50f00:::benchmark_data_smithi088_474635_object301:head# 2024-04-25T09:11:34.053 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.087 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:01d30b1e:::benchmark_data_smithi088_474635_object200:head# 2024-04-25T09:11:34.087 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.120 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0207f557:::benchmark_data_smithi088_474635_object278:head# 2024-04-25T09:11:34.120 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.154 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:029f4278:::benchmark_data_smithi088_474635_object32:head# 2024-04-25T09:11:34.154 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.187 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:02e035ca:::benchmark_data_smithi088_474635_object344:head# 2024-04-25T09:11:34.187 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.221 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:02f2bbca:::benchmark_data_smithi088_474635_object36:head# 2024-04-25T09:11:34.221 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.254 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:04aec993:::benchmark_data_smithi088_474635_object132:head# 2024-04-25T09:11:34.254 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.288 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:04e1c09c:::benchmark_data_smithi088_474635_object325:head# 2024-04-25T09:11:34.288 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.321 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:04ec75cb:::benchmark_data_smithi088_474635_object295:head# 2024-04-25T09:11:34.321 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.354 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:050c3199:::benchmark_data_smithi088_474635_object61:head# 2024-04-25T09:11:34.355 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.388 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:065efd69:::benchmark_data_smithi088_474635_object280:head# 2024-04-25T09:11:34.388 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.421 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:08630ca9:::benchmark_data_smithi088_474635_object14:head# 2024-04-25T09:11:34.422 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.455 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:086d2cde:::benchmark_data_smithi088_474635_object247:head# 2024-04-25T09:11:34.455 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.488 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:09bafaeb:::benchmark_data_smithi088_474635_object241:head# 2024-04-25T09:11:34.488 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.522 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:09fc5c5f:::benchmark_data_smithi088_474635_object6:head# 2024-04-25T09:11:34.522 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.555 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a5643a9:::benchmark_data_smithi088_474635_object158:head# 2024-04-25T09:11:34.555 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.589 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0a95b5cf:::benchmark_data_smithi088_474635_object260:head# 2024-04-25T09:11:34.589 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.622 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0e1b1263:::benchmark_data_smithi088_474635_object296:head# 2024-04-25T09:11:34.622 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.656 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:0f28134a:::benchmark_data_smithi088_474635_object98:head# 2024-04-25T09:11:34.656 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.689 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:10419da0:::benchmark_data_smithi088_474635_object148:head# 2024-04-25T09:11:34.689 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.722 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:10dac8b9:::benchmark_data_smithi088_474635_object63:head# 2024-04-25T09:11:34.723 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.806 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:10dca41f:::benchmark_data_smithi088_474635_object244:head# 2024-04-25T09:11:34.806 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.839 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1203d321:::benchmark_data_smithi088_474635_object193:head# 2024-04-25T09:11:34.839 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.873 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:123ce8ec:::benchmark_data_smithi088_474635_object203:head# 2024-04-25T09:11:34.873 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.906 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:12458a43:::benchmark_data_smithi088_474635_object121:head# 2024-04-25T09:11:34.907 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.940 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:125210df:::benchmark_data_smithi088_474635_object253:head# 2024-04-25T09:11:34.940 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:34.973 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:12d6f1fe:::benchmark_data_smithi088_474635_object113:head# 2024-04-25T09:11:34.973 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.007 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:12f2d4aa:::benchmark_data_smithi088_474635_object235:head# 2024-04-25T09:11:35.007 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.040 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:134281fb:::benchmark_data_smithi088_474635_object266:head# 2024-04-25T09:11:35.040 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.074 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:13537a3f:::benchmark_data_smithi088_474635_object57:head# 2024-04-25T09:11:35.074 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.107 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:13663042:::benchmark_data_smithi088_474635_object137:head# 2024-04-25T09:11:35.107 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.155 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:142c38e4:::benchmark_data_smithi088_474635_object367:head# 2024-04-25T09:11:35.155 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.233 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:14634413:::benchmark_data_smithi088_474635_object129:head# 2024-04-25T09:11:35.233 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.283 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:16f86b08:::benchmark_data_smithi088_474635_object178:head# 2024-04-25T09:11:35.283 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.316 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1723e09a:::benchmark_data_smithi088_474635_object147:head# 2024-04-25T09:11:35.316 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.350 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17c6165f:::benchmark_data_smithi088_474635_object232:head# 2024-04-25T09:11:35.350 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.383 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:17cea811:::benchmark_data_smithi088_474635_object223:head# 2024-04-25T09:11:35.383 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.416 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:180ab51c:::benchmark_data_smithi088_474635_object356:head# 2024-04-25T09:11:35.417 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.450 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:18bbb655:::benchmark_data_smithi088_474635_object65:head# 2024-04-25T09:11:35.450 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.483 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:195c99ac:::benchmark_data_smithi088_474635_object279:head# 2024-04-25T09:11:35.483 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.517 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1b89a1de:::benchmark_data_smithi088_474635_object68:head# 2024-04-25T09:11:35.517 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.550 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1cd35e0a:::benchmark_data_smithi088_474635_object261:head# 2024-04-25T09:11:35.550 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.584 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1cdffef9:::benchmark_data_smithi088_474635_object119:head# 2024-04-25T09:11:35.584 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.626 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1df233e4:::benchmark_data_smithi088_474635_object82:head# 2024-04-25T09:11:35.626 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.709 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:1f77643f:::benchmark_data_smithi088_474635_object197:head# 2024-04-25T09:11:35.710 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.760 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2105e4c1:::benchmark_data_smithi088_474635_object327:head# 2024-04-25T09:11:35.760 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.827 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:21389a19:::benchmark_data_smithi088_474635_object284:head# 2024-04-25T09:11:35.827 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.860 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2157c102:::benchmark_data_smithi088_474635_object263:head# 2024-04-25T09:11:35.860 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.894 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:216515d2:::benchmark_data_smithi088_474635_object190:head# 2024-04-25T09:11:35.894 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.927 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:21a850ef:::benchmark_data_smithi088_474635_object66:head# 2024-04-25T09:11:35.928 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.961 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:220dddb9:::benchmark_data_smithi088_474635_object103:head# 2024-04-25T09:11:35.961 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:35.994 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2314d873:::benchmark_data_smithi088_474635_object156:head# 2024-04-25T09:11:35.994 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.028 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:237fddad:::benchmark_data_smithi088_474635_object245:head# 2024-04-25T09:11:36.028 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.061 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2413ea50:::benchmark_data_smithi088_474635_object303:head# 2024-04-25T09:11:36.061 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.094 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2509de11:::benchmark_data_smithi088_474635_object124:head# 2024-04-25T09:11:36.095 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.128 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:25be00d9:::benchmark_data_smithi088_474635_object55:head# 2024-04-25T09:11:36.128 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.162 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:27662daf:::benchmark_data_smithi088_474635_object171:head# 2024-04-25T09:11:36.162 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.195 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:285bfb1b:::benchmark_data_smithi088_474635_object75:head# 2024-04-25T09:11:36.195 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.228 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:294df2b5:::benchmark_data_smithi088_474635_object194:head# 2024-04-25T09:11:36.228 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.262 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:29c92c62:::benchmark_data_smithi088_474635_object108:head# 2024-04-25T09:11:36.262 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.295 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2aa517bd:::benchmark_data_smithi088_474635_object104:head# 2024-04-25T09:11:36.295 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.337 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2aca7793:::benchmark_data_smithi088_474635_object160:head# 2024-04-25T09:11:36.337 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.370 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2b6ca883:::benchmark_data_smithi088_474635_object96:head# 2024-04-25T09:11:36.371 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.404 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2c329146:::benchmark_data_smithi088_474635_object312:head# 2024-04-25T09:11:36.404 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.437 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2c676fa6:::benchmark_data_smithi088_474635_object216:head# 2024-04-25T09:11:36.437 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.471 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2cb7968f:::benchmark_data_smithi088_474635_object225:head# 2024-04-25T09:11:36.471 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.504 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2ce995e4:::benchmark_data_smithi088_474635_object60:head# 2024-04-25T09:11:36.504 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.538 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2d705aec:::benchmark_data_smithi088_474635_object283:head# 2024-04-25T09:11:36.538 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.571 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2d7096ed:::benchmark_data_smithi088_474635_object362:head# 2024-04-25T09:11:36.571 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.605 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2e0a4ce6:::benchmark_data_smithi088_474635_object205:head# 2024-04-25T09:11:36.605 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.638 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2e2669dc:::benchmark_data_smithi088_474635_object143:head# 2024-04-25T09:11:36.638 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.672 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2e2679ac:::benchmark_data_smithi088_474635_object273:head# 2024-04-25T09:11:36.672 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.705 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2e5420b7:::benchmark_data_smithi088_474635_object272:head# 2024-04-25T09:11:36.705 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.738 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f121d11:::benchmark_data_smithi088_474635_object179:head# 2024-04-25T09:11:36.739 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.772 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f2ca407:::benchmark_data_smithi088_474635_object149:head# 2024-04-25T09:11:36.773 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.805 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f2eaa6f:::benchmark_data_smithi088_474635_object328:head# 2024-04-25T09:11:36.806 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.839 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f644dd2:::benchmark_data_smithi088_474635_object191:head# 2024-04-25T09:11:36.839 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.872 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:2f6a8b03:::benchmark_data_smithi088_474635_object360:head# 2024-04-25T09:11:36.872 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.906 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:319d1b1b:::benchmark_data_smithi088_474635_object281:head# 2024-04-25T09:11:36.906 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.939 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:32caca73:::benchmark_data_smithi088_474635_object308:head# 2024-04-25T09:11:36.939 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:36.973 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:34829fd5:::benchmark_data_smithi088_474635_object248:head# 2024-04-25T09:11:36.973 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.006 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:34fe1da9:::benchmark_data_smithi088_474635_object286:head# 2024-04-25T09:11:37.006 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.040 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:35e0e924:::benchmark_data_smithi088_474635_object198:head# 2024-04-25T09:11:37.040 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.073 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:369feb74:::benchmark_data_smithi088_474635_object111:head# 2024-04-25T09:11:37.073 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.106 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:36d320eb:::benchmark_data_smithi088_474635_object355:head# 2024-04-25T09:11:37.106 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.146 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3712cfef:::benchmark_data_smithi088_474635_object230:head# 2024-04-25T09:11:37.146 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.215 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:38365cc5:::benchmark_data_smithi088_474635_object242:head# 2024-04-25T09:11:37.215 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.265 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:38478e6e:::benchmark_data_smithi088_474635_object109:head# 2024-04-25T09:11:37.266 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.299 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:38fa84e3:::benchmark_data_smithi088_474635_object320:head# 2024-04-25T09:11:37.299 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.332 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:396a89f4:::benchmark_data_smithi088_474635_object45:head# 2024-04-25T09:11:37.332 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.366 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:399f132d:::benchmark_data_smithi088_474635_object307:head# 2024-04-25T09:11:37.366 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.399 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3a9610aa:::benchmark_data_smithi088_474635_object83:head# 2024-04-25T09:11:37.399 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.433 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3ba25773:::benchmark_data_smithi088_474635_object114:head# 2024-04-25T09:11:37.433 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.466 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3c5f6b6b:::benchmark_data_smithi088_474635_object2:head# 2024-04-25T09:11:37.466 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.500 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3d19860c:::benchmark_data_smithi088_474635_object251:head# 2024-04-25T09:11:37.500 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.533 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3d540a92:::benchmark_data_smithi088_474635_object229:head# 2024-04-25T09:11:37.533 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.567 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3e3230ca:::benchmark_data_smithi088_474635_object212:head# 2024-04-25T09:11:37.567 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.600 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:3febb60b:::benchmark_data_smithi088_474635_object50:head# 2024-04-25T09:11:37.600 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.633 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4056bc59:::benchmark_data_smithi088_474635_object240:head# 2024-04-25T09:11:37.634 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.717 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:40dc14fa:::benchmark_data_smithi088_474635_object180:head# 2024-04-25T09:11:37.718 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.784 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:410cb459:::benchmark_data_smithi088_474635_object151:head# 2024-04-25T09:11:37.784 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.860 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:418971ee:::benchmark_data_smithi088_474635_object174:head# 2024-04-25T09:11:37.860 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.885 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:428a380e:::benchmark_data_smithi088_474635_object293:head# 2024-04-25T09:11:37.885 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.943 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:428b1a67:::benchmark_data_smithi088_474635_object353:head# 2024-04-25T09:11:37.943 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:37.977 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:42922198:::benchmark_data_smithi088_474635_object145:head# 2024-04-25T09:11:37.977 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.010 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:42be703d:::benchmark_data_smithi088_474635_object165:head# 2024-04-25T09:11:38.011 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.044 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:465bb8d4:::benchmark_data_smithi088_474635_object365:head# 2024-04-25T09:11:38.044 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.077 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:47bb0a3c:::benchmark_data_smithi088_474635_object97:head# 2024-04-25T09:11:38.077 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.111 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:47f18852:::benchmark_data_smithi088_474635_object346:head# 2024-04-25T09:11:38.111 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.144 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4897fda7:::benchmark_data_smithi088_474635_object28:head# 2024-04-25T09:11:38.144 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.177 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:48f1a6ac:::benchmark_data_smithi088_474635_object154:head# 2024-04-25T09:11:38.178 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.211 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4a21aef9:::benchmark_data_smithi088_474635_object93:head# 2024-04-25T09:11:38.211 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.244 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4aa94f54:::benchmark_data_smithi088_474635_object224:head# 2024-04-25T09:11:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.278 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4bbd9dc9:::benchmark_data_smithi088_474635_object88:head# 2024-04-25T09:11:38.278 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.311 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c404371:::benchmark_data_smithi088_474635_object184:head# 2024-04-25T09:11:38.312 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.378 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4c6221a7:::benchmark_data_smithi088_474635_object126:head# 2024-04-25T09:11:38.378 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.412 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4f664fa4:::benchmark_data_smithi088_474635_object220:head# 2024-04-25T09:11:38.412 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.445 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4ff71927:::benchmark_data_smithi088_474635_object288:head# 2024-04-25T09:11:38.445 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.478 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:4ff83a37:::benchmark_data_smithi088_474635_object262:head# 2024-04-25T09:11:38.479 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.512 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:503dfc05:::benchmark_data_smithi088_474635_object23:head# 2024-04-25T09:11:38.512 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.545 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:506ac00a:::benchmark_data_smithi088_474635_object141:head# 2024-04-25T09:11:38.546 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.579 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:513ae8d4:::benchmark_data_smithi088_474635_object258:head# 2024-04-25T09:11:38.579 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.612 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:51b7b0b5:::benchmark_data_smithi088_474635_object290:head# 2024-04-25T09:11:38.613 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.646 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:52dad3af:::benchmark_data_smithi088_474635_object231:head# 2024-04-25T09:11:38.646 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.688 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:52fda022:::benchmark_data_smithi088_474635_object311:head# 2024-04-25T09:11:38.688 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.729 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:55034a6f:::benchmark_data_smithi088_474635_object316:head# 2024-04-25T09:11:38.729 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.763 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:56076090:::benchmark_data_smithi088_474635_object133:head# 2024-04-25T09:11:38.763 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.796 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:56538934:::benchmark_data_smithi088_474635_object5:head# 2024-04-25T09:11:38.797 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.830 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5721e5f8:::benchmark_data_smithi088_474635_object74:head# 2024-04-25T09:11:38.830 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.863 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5745116f:::benchmark_data_smithi088_474635_object233:head# 2024-04-25T09:11:38.863 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.897 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:579659b2:::benchmark_data_smithi088_474635_object215:head# 2024-04-25T09:11:38.897 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.930 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:57cb171b:::benchmark_data_smithi088_474635_object252:head# 2024-04-25T09:11:38.930 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.963 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5837f004:::benchmark_data_smithi088_474635_object208:head# 2024-04-25T09:11:38.964 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:38.997 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:59b91b69:::benchmark_data_smithi088_474635_object357:head# 2024-04-25T09:11:38.997 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.031 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5beb7e41:::benchmark_data_smithi088_474635_object72:head# 2024-04-25T09:11:39.031 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.064 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5d37aaa5:::benchmark_data_smithi088_474635_object289:head# 2024-04-25T09:11:39.064 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.097 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5d467473:::benchmark_data_smithi088_474635_object304:head# 2024-04-25T09:11:39.097 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.131 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5e2f8898:::benchmark_data_smithi088_474635_object213:head# 2024-04-25T09:11:39.131 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.215 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:5fda8027:::benchmark_data_smithi088_474635_object256:head# 2024-04-25T09:11:39.215 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.248 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:61ed8bbc:::benchmark_data_smithi088_474635_object116:head# 2024-04-25T09:11:39.248 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.281 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:620fc497:::benchmark_data_smithi088_474635_object257:head# 2024-04-25T09:11:39.281 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.315 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:62363145:::benchmark_data_smithi088_474635_object125:head# 2024-04-25T09:11:39.315 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.348 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:625044ca:::benchmark_data_smithi088_474635_object29:head# 2024-04-25T09:11:39.348 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.382 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:637d890b:::benchmark_data_smithi088_474635_object285:head# 2024-04-25T09:11:39.382 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.415 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:63bee89e:::benchmark_data_smithi088_474635_object195:head# 2024-04-25T09:11:39.415 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.449 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:673a8823:::benchmark_data_smithi088_474635_object71:head# 2024-04-25T09:11:39.449 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.482 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:67c9fe8c:::benchmark_data_smithi088_474635_object313:head# 2024-04-25T09:11:39.482 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.516 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:686141c3:::benchmark_data_smithi088_474635_object352:head# 2024-04-25T09:11:39.516 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.549 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:697aefc7:::benchmark_data_smithi088_474635_object142:head# 2024-04-25T09:11:39.549 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.582 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6a2142c4:::benchmark_data_smithi088_474635_object222:head# 2024-04-25T09:11:39.582 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.616 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6a582a71:::benchmark_data_smithi088_474635_object21:head# 2024-04-25T09:11:39.616 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.662 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6ac0d24f:::benchmark_data_smithi088_474635_object211:head# 2024-04-25T09:11:39.662 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.759 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6b27391b:::benchmark_data_smithi088_474635_object90:head# 2024-04-25T09:11:39.759 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.842 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6b2f1bd4:::benchmark_data_smithi088_474635_object16:head# 2024-04-25T09:11:39.842 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.876 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6bdc7245:::benchmark_data_smithi088_474635_object89:head# 2024-04-25T09:11:39.876 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.909 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6c85a23d:::benchmark_data_smithi088_474635_object130:head# 2024-04-25T09:11:39.909 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.943 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6c97d0c1:::benchmark_data_smithi088_474635_object44:head# 2024-04-25T09:11:39.943 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:39.976 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6dd55281:::benchmark_data_smithi088_474635_object333:head# 2024-04-25T09:11:39.976 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.010 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6eaeb124:::benchmark_data_smithi088_474635_object3:head# 2024-04-25T09:11:40.010 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.043 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:6ecb117e:::benchmark_data_smithi088_474635_object182:head# 2024-04-25T09:11:40.043 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.076 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7029915e:::benchmark_data_smithi088_474635_object217:head# 2024-04-25T09:11:40.076 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.110 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:70425cf6:::benchmark_data_smithi088_474635_object35:head# 2024-04-25T09:11:40.110 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.143 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:70874157:::benchmark_data_smithi088_474635_object170:head# 2024-04-25T09:11:40.143 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.177 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:725e36bc:::benchmark_data_smithi088_474635_object269:head# 2024-04-25T09:11:40.177 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.210 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:72b36e3d:::benchmark_data_smithi088_474635_object153:head# 2024-04-25T09:11:40.210 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.244 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:738ae35b:::benchmark_data_smithi088_474635_object122:head# 2024-04-25T09:11:40.244 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.269 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:74adfe87:::benchmark_data_smithi088_474635_object302:head# 2024-04-25T09:11:40.269 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.311 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:758c96f1:::benchmark_data_smithi088_474635_object214:head# 2024-04-25T09:11:40.311 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.352 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:75e520f7:::benchmark_data_smithi088_474635_object169:head# 2024-04-25T09:11:40.353 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.394 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:76faa396:::benchmark_data_smithi088_474635_object128:head# 2024-04-25T09:11:40.394 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.436 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:782b27ad:::benchmark_data_smithi088_474635_object350:head# 2024-04-25T09:11:40.436 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.478 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7a97a86c:::benchmark_data_smithi088_474635_object26:head# 2024-04-25T09:11:40.478 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.520 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7abe2685:::benchmark_data_smithi088_474635_object299:head# 2024-04-25T09:11:40.520 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.561 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7bc6a2b5:::benchmark_data_smithi088_474635_object347:head# 2024-04-25T09:11:40.561 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.603 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7c492c96:::benchmark_data_smithi088_474635_object228:head# 2024-04-25T09:11:40.603 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.692 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7ce6829c:::benchmark_data_smithi088_474635_object0:head# 2024-04-25T09:11:40.692 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.770 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7d7fe9bd:::benchmark_data_smithi088_474635_object106:head# 2024-04-25T09:11:40.771 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.846 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7e52051d:::benchmark_data_smithi088_474635_object24:head# 2024-04-25T09:11:40.846 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.935 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7f01f963:::benchmark_data_smithi088_474635_object25:head# 2024-04-25T09:11:40.935 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:40.997 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7f887662:::benchmark_data_smithi088_474635_object64:head# 2024-04-25T09:11:40.997 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.030 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:7f8a93a8:::benchmark_data_smithi088_474635_object139:head# 2024-04-25T09:11:41.030 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.064 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8072f058:::benchmark_data_smithi088_474635_object172:head# 2024-04-25T09:11:41.064 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.097 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:80eaa411:::benchmark_data_smithi088_474635_object177:head# 2024-04-25T09:11:41.097 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.131 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8154841a:::benchmark_data_smithi088_474635_object227:head# 2024-04-25T09:11:41.131 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.231 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8154f227:::benchmark_data_smithi088_474635_object276:head# 2024-04-25T09:11:41.231 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.290 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8191b4d4:::benchmark_data_smithi088_474635_object332:head# 2024-04-25T09:11:41.290 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.323 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:82d242f9:::benchmark_data_smithi088_474635_object62:head# 2024-04-25T09:11:41.323 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.356 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:82fd563c:::benchmark_data_smithi088_474635_object183:head# 2024-04-25T09:11:41.357 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.390 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8308ab9e:::benchmark_data_smithi088_474635_object41:head# 2024-04-25T09:11:41.390 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.432 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8559d1df:::benchmark_data_smithi088_474635_object112:head# 2024-04-25T09:11:41.432 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.465 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:857941d3:::benchmark_data_smithi088_474635_object168:head# 2024-04-25T09:11:41.465 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.499 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:85d243ce:::benchmark_data_smithi088_474635_object42:head# 2024-04-25T09:11:41.499 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.532 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8662071c:::benchmark_data_smithi088_474635_object39:head# 2024-04-25T09:11:41.532 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.565 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8663fd45:::benchmark_data_smithi088_474635_object4:head# 2024-04-25T09:11:41.566 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.607 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:86798526:::benchmark_data_smithi088_474635_object300:head# 2024-04-25T09:11:41.607 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.641 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:86958939:::benchmark_data_smithi088_474635_object255:head# 2024-04-25T09:11:41.641 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.725 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:86d5de05:::benchmark_data_smithi088_474635_object53:head# 2024-04-25T09:11:41.725 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.800 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:87295d2d:::benchmark_data_smithi088_474635_object340:head# 2024-04-25T09:11:41.800 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.859 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:895bbe02:::benchmark_data_smithi088_474635_object204:head# 2024-04-25T09:11:41.859 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.922 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8c544bab:::benchmark_data_smithi088_474635_object187:head# 2024-04-25T09:11:41.922 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:41.967 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8dc506d9:::benchmark_data_smithi088_474635_object309:head# 2024-04-25T09:11:41.967 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.001 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8e17a3ba:::benchmark_data_smithi088_474635_object30:head# 2024-04-25T09:11:42.001 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.042 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8e193346:::benchmark_data_smithi088_474635_object249:head# 2024-04-25T09:11:42.043 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.084 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8e813d25:::benchmark_data_smithi088_474635_object162:head# 2024-04-25T09:11:42.084 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.126 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8f067153:::benchmark_data_smithi088_474635_object236:head# 2024-04-25T09:11:42.126 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.168 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8f50ce47:::benchmark_data_smithi088_474635_object48:head# 2024-04-25T09:11:42.168 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.210 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:8f535138:::benchmark_data_smithi088_474635_object274:head# 2024-04-25T09:11:42.210 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.268 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:91e6cc24:::benchmark_data_smithi088_474635_object123:head# 2024-04-25T09:11:42.268 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.310 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9201d91f:::benchmark_data_smithi088_474635_object10:head# 2024-04-25T09:11:42.310 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.352 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92193071:::benchmark_data_smithi088_474635_object173:head# 2024-04-25T09:11:42.352 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.393 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:923c5ad1:::benchmark_data_smithi088_474635_object254:head# 2024-04-25T09:11:42.394 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.435 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:92759122:::benchmark_data_smithi088_474635_object206:head# 2024-04-25T09:11:42.435 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.477 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9312eb3e:::benchmark_data_smithi088_474635_object131:head# 2024-04-25T09:11:42.477 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.519 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9357f6c0:::benchmark_data_smithi088_474635_object239:head# 2024-04-25T09:11:42.519 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.552 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:937bf92d:::benchmark_data_smithi088_474635_object334:head# 2024-04-25T09:11:42.552 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.586 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9439d424:::benchmark_data_smithi088_474635_object270:head# 2024-04-25T09:11:42.586 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.619 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:957c4957:::benchmark_data_smithi088_474635_object292:head# 2024-04-25T09:11:42.619 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.644 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9605c18c:::benchmark_data_smithi088_474635_object84:head# 2024-04-25T09:11:42.644 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.669 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:97dc4359:::benchmark_data_smithi088_474635_object91:head# 2024-04-25T09:11:42.669 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.703 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:989d0658:::benchmark_data_smithi088_474635_object47:head# 2024-04-25T09:11:42.703 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.736 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:99596455:::benchmark_data_smithi088_474635_object18:head# 2024-04-25T09:11:42.736 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.778 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9a5c4743:::benchmark_data_smithi088_474635_object43:head# 2024-04-25T09:11:42.778 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.820 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9a719784:::benchmark_data_smithi088_474635_object12:head# 2024-04-25T09:11:42.820 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.862 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9acc2858:::benchmark_data_smithi088_474635_object338:head# 2024-04-25T09:11:42.862 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.903 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9ae892d9:::benchmark_data_smithi088_474635_object335:head# 2024-04-25T09:11:42.903 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.945 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9c214df4:::benchmark_data_smithi088_474635_object339:head# 2024-04-25T09:11:42.945 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:42.979 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9cf0af18:::benchmark_data_smithi088_474635_object146:head# 2024-04-25T09:11:42.979 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.012 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:9d679c58:::benchmark_data_smithi088_474635_object155:head# 2024-04-25T09:11:43.012 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.054 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a154514b:::benchmark_data_smithi088_474635_object243:head# 2024-04-25T09:11:43.054 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.096 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a1c6e016:::benchmark_data_smithi088_474635_object199:head# 2024-04-25T09:11:43.096 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.146 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a22943b6:::benchmark_data_smithi088_474635_object287:head# 2024-04-25T09:11:43.146 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.213 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a2867657:::benchmark_data_smithi088_474635_object87:head# 2024-04-25T09:11:43.213 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.255 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a2dbd594:::benchmark_data_smithi088_474635_object310:head# 2024-04-25T09:11:43.255 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.296 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a3d32bff:::benchmark_data_smithi088_474635_object101:head# 2024-04-25T09:11:43.297 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.338 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a43a8dab:::benchmark_data_smithi088_474635_object86:head# 2024-04-25T09:11:43.338 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.380 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a5753aec:::benchmark_data_smithi088_474635_object31:head# 2024-04-25T09:11:43.380 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.422 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a6b94731:::benchmark_data_smithi088_474635_object161:head# 2024-04-25T09:11:43.422 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.464 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a74259fa:::benchmark_data_smithi088_474635_object166:head# 2024-04-25T09:11:43.464 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.505 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a916d9c4:::benchmark_data_smithi088_474635_object341:head# 2024-04-25T09:11:43.505 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.547 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a92c6ab4:::benchmark_data_smithi088_474635_object102:head# 2024-04-25T09:11:43.547 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.589 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:a9ada565:::benchmark_data_smithi088_474635_object134:head# 2024-04-25T09:11:43.589 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.631 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:aa9ac8f2:::benchmark_data_smithi088_474635_object219:head# 2024-04-25T09:11:43.631 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.706 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:aac5607b:::benchmark_data_smithi088_474635_object321:head# 2024-04-25T09:11:43.706 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.773 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab2bd6b0:::benchmark_data_smithi088_474635_object361:head# 2024-04-25T09:11:43.773 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.848 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab5f28f6:::benchmark_data_smithi088_474635_object34:head# 2024-04-25T09:11:43.849 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.890 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ab8ca43f:::benchmark_data_smithi088_474635_object326:head# 2024-04-25T09:11:43.890 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.932 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:acbd0c95:::benchmark_data_smithi088_474635_object138:head# 2024-04-25T09:11:43.932 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:43.974 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ad03517a:::benchmark_data_smithi088_474635_object267:head# 2024-04-25T09:11:43.974 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.016 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ad8f42e7:::benchmark_data_smithi088_474635_object330:head# 2024-04-25T09:11:44.016 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.057 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b3161fff:::benchmark_data_smithi088_474635_object94:head# 2024-04-25T09:11:44.057 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.099 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b34353cf:::benchmark_data_smithi088_474635_object185:head# 2024-04-25T09:11:44.099 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.141 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b3d5afaf:::benchmark_data_smithi088_474635_object192:head# 2024-04-25T09:11:44.141 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.183 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b4a70d6d:::benchmark_data_smithi088_474635_object136:head# 2024-04-25T09:11:44.183 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.225 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b4aa86e1:::benchmark_data_smithi088_474635_object115:head# 2024-04-25T09:11:44.225 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.267 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b4efc706:::benchmark_data_smithi088_474635_object150:head# 2024-04-25T09:11:44.267 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.309 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b6a56dd3:::benchmark_data_smithi088_474635_object221:head# 2024-04-25T09:11:44.309 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.350 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b6b27c77:::benchmark_data_smithi088_474635_object364:head# 2024-04-25T09:11:44.350 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.392 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b6d0fb40:::benchmark_data_smithi088_474635_object366:head# 2024-04-25T09:11:44.392 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.434 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b8518f13:::benchmark_data_smithi088_474635_object17:head# 2024-04-25T09:11:44.434 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.476 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:11:44.476 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.517 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ba19a0a4:::benchmark_data_smithi088_474635_object343:head# 2024-04-25T09:11:44.517 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.559 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bacf247d:::benchmark_data_smithi088_474635_object342:head# 2024-04-25T09:11:44.559 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.601 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bd320262:::benchmark_data_smithi088_474635_object349:head# 2024-04-25T09:11:44.601 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.643 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bd48bd40:::benchmark_data_smithi088_474635_object22:head# 2024-04-25T09:11:44.643 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.685 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bd5991cf:::benchmark_data_smithi088_474635_object33:head# 2024-04-25T09:11:44.685 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.726 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bec41a8c:::benchmark_data_smithi088_474635_object85:head# 2024-04-25T09:11:44.727 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.768 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:bf6a0f1a:::benchmark_data_smithi088_474635_object13:head# 2024-04-25T09:11:44.768 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.810 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c0e5028f:::benchmark_data_smithi088_474635_object324:head# 2024-04-25T09:11:44.810 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.852 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c11e469d:::benchmark_data_smithi088_474635_object77:head# 2024-04-25T09:11:44.852 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.893 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c136c80d:::benchmark_data_smithi088_474635_object207:head# 2024-04-25T09:11:44.893 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.935 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c26ea520:::benchmark_data_smithi088_474635_object92:head# 2024-04-25T09:11:44.935 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:44.994 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c40ced02:::benchmark_data_smithi088_474635_object277:head# 2024-04-25T09:11:44.994 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.035 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c47f6c72:::benchmark_data_smithi088_474635_object305:head# 2024-04-25T09:11:45.035 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.077 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c617de94:::benchmark_data_smithi088_474635_object291:head# 2024-04-25T09:11:45.077 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.119 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c6181fd8:::benchmark_data_smithi088_474635_object105:head# 2024-04-25T09:11:45.119 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.161 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c6992680:::benchmark_data_smithi088_474635_object268:head# 2024-04-25T09:11:45.161 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.236 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c714c83e:::benchmark_data_smithi088_474635_object202:head# 2024-04-25T09:11:45.236 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.278 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c7263d37:::benchmark_data_smithi088_474635_object189:head# 2024-04-25T09:11:45.278 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.320 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c7c7f9c1:::benchmark_data_smithi088_474635_object294:head# 2024-04-25T09:11:45.320 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.353 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c80fc382:::benchmark_data_smithi088_474635_object40:head# 2024-04-25T09:11:45.353 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.387 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c98d5fcc:::benchmark_data_smithi088_474635_object19:head# 2024-04-25T09:11:45.387 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.412 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:c9e15eb4:::benchmark_data_smithi088_474635_object9:head# 2024-04-25T09:11:45.412 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.445 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ca7c35cc:::benchmark_data_smithi088_474635_object73:head# 2024-04-25T09:11:45.445 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.479 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ca86eae0:::benchmark_data_smithi088_474635_object100:head# 2024-04-25T09:11:45.479 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.554 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:caaa7e6f:::benchmark_data_smithi088_474635_object348:head# 2024-04-25T09:11:45.554 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.595 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cbb7c498:::benchmark_data_smithi088_474635_object20:head# 2024-04-25T09:11:45.596 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.637 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cbfcab2c:::benchmark_data_smithi088_474635_object117:head# 2024-04-25T09:11:45.637 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.705 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ccb4b73f:::benchmark_data_smithi088_474635_object59:head# 2024-04-25T09:11:45.705 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.780 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cd545eae:::benchmark_data_smithi088_474635_object157:head# 2024-04-25T09:11:45.780 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.830 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cd692c8e:::benchmark_data_smithi088_474635_object81:head# 2024-04-25T09:11:45.830 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.889 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:cdf279d0:::benchmark_data_smithi088_474635_object358:head# 2024-04-25T09:11:45.889 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.930 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ce4241dd:::benchmark_data_smithi088_474635_object314:head# 2024-04-25T09:11:45.930 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:45.972 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ce7922f5:::benchmark_data_smithi088_474635_object345:head# 2024-04-25T09:11:45.972 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.022 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d06c083a:::benchmark_data_smithi088_474635_object275:head# 2024-04-25T09:11:46.022 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.064 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d233b28d:::benchmark_data_smithi088_474635_object317:head# 2024-04-25T09:11:46.064 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.106 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d31f9487:::benchmark_data_smithi088_474635_object56:head# 2024-04-25T09:11:46.106 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.148 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d3269e90:::benchmark_data_smithi088_474635_object140:head# 2024-04-25T09:11:46.148 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.190 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d398c90e:::benchmark_data_smithi088_474635_object237:head# 2024-04-25T09:11:46.190 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.231 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d40e39ec:::benchmark_data_smithi088_474635_object95:head# 2024-04-25T09:11:46.231 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.273 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d462d81d:::benchmark_data_smithi088_474635_object135:head# 2024-04-25T09:11:46.273 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.315 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d4ee28e3:::benchmark_data_smithi088_474635_object337:head# 2024-04-25T09:11:46.315 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.357 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d572577d:::benchmark_data_smithi088_474635_object164:head# 2024-04-25T09:11:46.357 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.398 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d5c8e00e:::benchmark_data_smithi088_474635_object336:head# 2024-04-25T09:11:46.398 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.440 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d5d3de0c:::benchmark_data_smithi088_474635_object38:head# 2024-04-25T09:11:46.440 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.482 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d7234313:::benchmark_data_smithi088_474635_object234:head# 2024-04-25T09:11:46.482 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.524 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d741ddf1:::benchmark_data_smithi088_474635_object37:head# 2024-04-25T09:11:46.524 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.566 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:d8774faa:::benchmark_data_smithi088_474635_object306:head# 2024-04-25T09:11:46.566 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.607 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dad3e05a:::benchmark_data_smithi088_474635_object318:head# 2024-04-25T09:11:46.607 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.649 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:db51ceaf:::benchmark_data_smithi088_474635_object167:head# 2024-04-25T09:11:46.649 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.700 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dbcddc20:::benchmark_data_smithi088_474635_object144:head# 2024-04-25T09:11:46.700 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.741 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dc239daa:::benchmark_data_smithi088_474635_object76:head# 2024-04-25T09:11:46.742 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.783 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dc4dbfb6:::benchmark_data_smithi088_474635_object120:head# 2024-04-25T09:11:46.783 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.825 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dcbc5866:::benchmark_data_smithi088_474635_object110:head# 2024-04-25T09:11:46.825 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.867 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dcf7bf9c:::benchmark_data_smithi088_474635_object186:head# 2024-04-25T09:11:46.867 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.908 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd72c389:::benchmark_data_smithi088_474635_object79:head# 2024-04-25T09:11:46.908 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.950 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dd7f00f3:::benchmark_data_smithi088_474635_object322:head# 2024-04-25T09:11:46.950 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:46.992 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ddbf17cd:::benchmark_data_smithi088_474635_object58:head# 2024-04-25T09:11:46.992 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.076 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:de74aab6:::benchmark_data_smithi088_474635_object107:head# 2024-04-25T09:11:47.076 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.117 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dfcc03c2:::benchmark_data_smithi088_474635_object127:head# 2024-04-25T09:11:47.117 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.159 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:dfe8a4f8:::benchmark_data_smithi088_474635_object265:head# 2024-04-25T09:11:47.160 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.234 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e29d67fb:::benchmark_data_smithi088_474635_object69:head# 2024-04-25T09:11:47.235 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.276 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e2abbb39:::benchmark_data_smithi088_474635_object298:head# 2024-04-25T09:11:47.276 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.318 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e3557d2e:::benchmark_data_smithi088_474635_object8:head# 2024-04-25T09:11:47.318 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.360 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e52260c6:::benchmark_data_smithi088_474635_object259:head# 2024-04-25T09:11:47.360 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.402 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e691a3f1:::benchmark_data_smithi088_474635_object238:head# 2024-04-25T09:11:47.402 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.444 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e7e60cd1:::benchmark_data_smithi088_474635_object218:head# 2024-04-25T09:11:47.444 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.485 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:e92137d0:::benchmark_data_smithi088_474635_object46:head# 2024-04-25T09:11:47.485 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.527 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ea1fc82e:::benchmark_data_smithi088_474635_object271:head# 2024-04-25T09:11:47.527 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.569 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ea235d8b:::benchmark_data_smithi088_474635_object329:head# 2024-04-25T09:11:47.569 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.610 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ea5df4a2:::benchmark_data_smithi088_474635_object354:head# 2024-04-25T09:11:47.611 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.660 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:eae3ed2d:::benchmark_data_smithi088_474635_object196:head# 2024-04-25T09:11:47.660 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.736 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ebf9a4c3:::benchmark_data_smithi088_474635_object319:head# 2024-04-25T09:11:47.736 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.820 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ed8f3a1b:::benchmark_data_smithi088_474635_object80:head# 2024-04-25T09:11:47.820 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.862 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ee957fda:::benchmark_data_smithi088_474635_object51:head# 2024-04-25T09:11:47.862 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.903 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ef265467:::benchmark_data_smithi088_474635_object15:head# 2024-04-25T09:11:47.904 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.945 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f0514099:::benchmark_data_smithi088_474635_object11:head# 2024-04-25T09:11:47.945 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:47.987 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f06dd765:::benchmark_data_smithi088_474635_object282:head# 2024-04-25T09:11:47.987 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.029 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f0987f6b:::benchmark_data_smithi088_474635_object264:head# 2024-04-25T09:11:48.029 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.062 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f0cdbd0b:::benchmark_data_smithi088_474635_object176:head# 2024-04-25T09:11:48.062 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.096 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f225e4b0:::benchmark_data_smithi088_474635_object118:head# 2024-04-25T09:11:48.096 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.121 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f2563bb4:::benchmark_data_smithi088_474635_object226:head# 2024-04-25T09:11:48.121 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.146 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f2aaec93:::benchmark_data_smithi088_474635_object152:head# 2024-04-25T09:11:48.146 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.179 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f2d5ca00:::benchmark_data_smithi088_474635_object159:head# 2024-04-25T09:11:48.179 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.213 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f3cd498d:::benchmark_data_smithi088_474635_object78:head# 2024-04-25T09:11:48.213 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.246 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f3f30a65:::benchmark_data_smithi088_474635_object163:head# 2024-04-25T09:11:48.246 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.288 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f4caeb31:::benchmark_data_smithi088_474635_object315:head# 2024-04-25T09:11:48.288 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.330 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f5425fd8:::benchmark_data_smithi088_474635_object27:head# 2024-04-25T09:11:48.330 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.372 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f560643f:::benchmark_data_smithi088_474635_object363:head# 2024-04-25T09:11:48.372 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.413 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f7085f7a:::benchmark_data_smithi088_474635_object181:head# 2024-04-25T09:11:48.414 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.447 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f72b2424:::benchmark_data_smithi088_474635_object331:head# 2024-04-25T09:11:48.447 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.480 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f76b0004:::benchmark_data_smithi088_474635_object188:head# 2024-04-25T09:11:48.480 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.514 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f82f89f0:::benchmark_data_smithi088_474635_object1:head# 2024-04-25T09:11:48.514 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.547 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f8b0d0ea:::benchmark_data_smithi088_474635_object99:head# 2024-04-25T09:11:48.547 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.581 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f8c1439b:::benchmark_data_smithi088_474635_object52:head# 2024-04-25T09:11:48.581 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.622 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:f8ca900e:::benchmark_data_smithi088_474635_object246:head# 2024-04-25T09:11:48.623 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.664 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fc5127b1:::benchmark_data_smithi088_474635_object210:head# 2024-04-25T09:11:48.664 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.706 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fc806264:::benchmark_data_smithi088_474635_object359:head# 2024-04-25T09:11:48.706 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.748 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fcf4dc66:::benchmark_data_smithi088_474635_object209:head# 2024-04-25T09:11:48.748 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.790 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fe726c16:::benchmark_data_smithi088_474635_object49:head# 2024-04-25T09:11:48.790 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.832 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:fe7aee35:::benchmark_data_smithi088_474635_object54:head# 2024-04-25T09:11:48.832 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.873 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ff268577:::benchmark_data_smithi088_474635_object7:head# 2024-04-25T09:11:48.874 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.915 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ff457b2a:::benchmark_data_smithi088_474635_object323:head# 2024-04-25T09:11:48.915 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.957 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ff7af289:::benchmark_data_smithi088_474635_object250:head# 2024-04-25T09:11:48.957 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:48.999 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ffa0454c:::benchmark_data_smithi088_474635_object175:head# 2024-04-25T09:11:48.999 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:49.041 INFO:tasks.workunit.client.0.smithi088.stdout:Write #1:ffc40287:::benchmark_data_smithi088_474635_object351:head# 2024-04-25T09:11:49.041 INFO:tasks.workunit.client.0.smithi088.stdout:snapset 0=[]:{} 2024-04-25T09:11:49.103 INFO:tasks.workunit.client.0.smithi088.stdout:write_pg epoch 9 info 1.0( v 9'369 (0'0,9'369] local-lis/les=7/8 n=369 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:11:49.168 INFO:tasks.workunit.client.0.smithi088.stdout:Import successful 2024-04-25T09:11:49.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:193: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-04-25T09:11:49.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:11:49.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:11:49.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:11:49.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:11:49.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:11:49.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:11:49.707 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:11:49.708 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:11:49.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:11:49.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:49.709 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:11:49.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:11:49.711 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:11:49.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:11:49.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=96ba146e-cb5e-4763-b54a-efd66661b729 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451260/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:11:49.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:11:49.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:11:49.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:11:49.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:11:49.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:11:49.748 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:49.758+0000 7fc5f8ec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:49.751 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:49.762+0000 7fc5f8ec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:49.753 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:49.762+0000 7fc5f8ec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:50.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:11:50.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:11:50.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:11:50.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:11:50.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:11:50.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:50.149 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:11:50.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:11:50.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:50.150 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:50.323 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:50.334+0000 7fc5f8ec87c0 -1 Falling back to public interface 2024-04-25T09:11:50.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:51.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:51.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:51.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:11:51.568 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:11:51.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:51.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:51.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:52.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:52.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:52.978 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:11:52.978 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:11:52.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:52.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:53.371 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:11:53.382+0000 7fc5f8ec87c0 -1 osd.0 18 log_to_monitors true 2024-04-25T09:11:53.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:54.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:54.505 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:54.505 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:11:54.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:11:54.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:54.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:54.935 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [12,18) [v2:127.0.0.1:6802/3452104383,v1:127.0.0.1:6803/3452104383] [v2:127.0.0.1:6804/3452104383,v1:127.0.0.1:6805/3452104383] exists,up d4271cdc-3827-4921-8177-f9d1daa5379e 2024-04-25T09:11:54.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:11:54.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:11:54.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:11:54.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:195: TEST_import_after_split: wait_for_clean 2024-04-25T09:11:54.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:11:54.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:11:54.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:11:54.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:11:54.937 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:11:54.937 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:11:54.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:11:54.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:11:54.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:11:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:11:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:11:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:11:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:11:55.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:11:55.039 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:11:55.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:11:55.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:11:55.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:11:55.521 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:11:55.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313219 2024-04-25T09:11:55.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313219 2024-04-25T09:11:55.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-90194313219' 2024-04-25T09:11:55.695 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:11:55.695 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-90194313219 2024-04-25T09:11:55.696 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:11:55.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:11:55.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-90194313219 2024-04-25T09:11:55.697 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:11:55.698 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 90194313219 2024-04-25T09:11:55.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313219 2024-04-25T09:11:55.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 90194313219' 2024-04-25T09:11:55.699 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:11:56.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 90194313219 2024-04-25T09:11:56.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:11:57.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:11:57.114 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:11:57.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313219 -lt 90194313219 2024-04-25T09:11:57.525 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:11:57.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:11:57.526 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:11:58.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:11:58.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:11:58.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:11:58.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:11:58.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:11:58.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:11:58.100 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:11:58.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:11:58.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:11:58.490 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:11:58.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:11:58.491 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:11:59.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:11:59.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:11:59.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:11:59.027 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:11:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:11:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:11:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:11:59.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:11:59.028 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:11:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:11:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:11:59.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:11:59.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:11:59.135 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:11:59.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:11:59.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:11:59.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:11:59.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:11:59.137 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:11:59.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:11:59.138 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:11:59.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:11:59.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:11:59.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:11:59.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:11:59.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:11:59.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:11:59.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:59.151 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:11:59.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451260 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/pg-split-merge 0 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:11:59.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:11:59.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:11:59.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:11:59.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:11:59.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:11:59.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:11:59.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:11:59.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:11:59.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:11:59.156 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:11:59.156 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:11:59.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:11:59.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:11:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:11:59.158 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:11:59.158 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:11:59.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:11:59.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:11:59.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:11:59.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:59.161 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451260 2024-04-25T09:11:59.161 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451260 2024-04-25T09:11:59.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:11:59.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:11:59.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:11:59.163 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:11:59.164 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:11:59.173 INFO:tasks.workunit:Running workunit osd/repeer-on-acting-back.sh... 2024-04-25T09:11:59.174 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/repeer-on-acting-back.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh 2024-04-25T09:11:59.228 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:11:59.234 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/repeer-on-acting-back 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:22: run: local dir=td/repeer-on-acting-back 2024-04-25T09:11:59.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:23: run: shift 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: export poolname=test 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: poolname=test 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: export testobjects=100 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: testobjects=100 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: export loglen=12 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: loglen=12 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: expr 12 / 2 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: export trim=6 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: trim=6 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: CEPH_MON=127.0.0.1:7115 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:30: run: export CEPH_ARGS 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: uuidgen 2024-04-25T09:11:59.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: CEPH_ARGS+='--fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none ' 2024-04-25T09:11:59.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T09:11:59.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:34: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:11:59.237 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:36: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:11:59.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: set 2024-04-25T09:11:59.238 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: local funcs=TEST_repeer_on_down_acting_member_coming_back 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:39: run: for func in $funcs 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:40: run: setup td/repeer-on-acting-back 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repeer-on-acting-back 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repeer-on-acting-back 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:11:59.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:11:59.240 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:11:59.240 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:11:59.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:11:59.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:11:59.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:11:59.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:11:59.242 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:11:59.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:11:59.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:11:59.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:11:59.244 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:11:59.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:11:59.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:11:59.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:11:59.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:11:59.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:59.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:11:59.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.478073 2024-04-25T09:11:59.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:11:59.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:11:59.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repeer-on-acting-back 2024-04-25T09:11:59.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:11:59.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:59.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:11:59.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.478073 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repeer-on-acting-back 1' TERM HUP INT 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:41: run: TEST_repeer_on_down_acting_member_coming_back td/repeer-on-acting-back 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:48: TEST_repeer_on_down_acting_member_coming_back: local dir=td/repeer-on-acting-back 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:49: TEST_repeer_on_down_acting_member_coming_back: local dummyfile=/etc/fstab 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:51: TEST_repeer_on_down_acting_member_coming_back: local num_osds=6 2024-04-25T09:11:59.251 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: expr 6 - 1 2024-04-25T09:11:59.252 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: seq 0 5 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: local 'osds=0 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:53: TEST_repeer_on_down_acting_member_coming_back: run_mon td/repeer-on-acting-back a 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repeer-on-acting-back 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:11:59.253 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repeer-on-acting-back/a 2024-04-25T09:11:59.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repeer-on-acting-back/a --run-dir=td/repeer-on-acting-back 2024-04-25T09:11:59.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:11:59.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:11:59.726 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:11:59.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:11:59.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:59.726 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:11:59.727 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:11:59.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repeer-on-acting-back/a '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --mon-cluster-log-file=td/repeer-on-acting-back/log --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:11:59.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:11:59.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:11:59.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:11:59.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:11:59.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:11:59.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:11:59.972 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:11:59.973 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:11:59.973 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:11:59.973 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:11:59.974 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:59.974 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:11:59.974 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.478073/ceph-mon.a.asok 2024-04-25T09:11:59.974 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:11:59.974 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.478073/ceph-mon.a.asok config get fsid 2024-04-25T09:12:00.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:12:00.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:12:00.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:12:00.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:12:00.087 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:12:00.087 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:12:00.087 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:12:00.088 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:12:00.088 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:00.088 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:00.088 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:12:00.088 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.478073/ceph-mon.a.asok 2024-04-25T09:12:00.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:12:00.088 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.478073/ceph-mon.a.asok config get mon_host 2024-04-25T09:12:00.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:54: TEST_repeer_on_down_acting_member_coming_back: run_mgr td/repeer-on-acting-back x 2024-04-25T09:12:00.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repeer-on-acting-back 2024-04-25T09:12:00.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:12:00.188 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:12:00.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:12:00.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repeer-on-acting-back/x 2024-04-25T09:12:00.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:12:01.128 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:12:01.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:12:01.129 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:12:01.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:12:01.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:01.129 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:12:01.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:12:01.130 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:12:01.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repeer-on-acting-back/x '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:12:01.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:12:01.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 0 2024-04-25T09:12:01.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/0 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/0' 2024-04-25T09:12:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/0/journal' 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:12:01.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:12:01.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/0 2024-04-25T09:12:01.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:12:01.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=208322d1-3dfc-4908-8081-a6defff8b8e6 2024-04-25T09:12:01.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 208322d1-3dfc-4908-8081-a6defff8b8e6' 2024-04-25T09:12:01.209 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 208322d1-3dfc-4908-8081-a6defff8b8e6 2024-04-25T09:12:01.209 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:12:01.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhHipmZvB6DhAAZLEHzvA41aXP2Pn/KTfWlg== 2024-04-25T09:12:01.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhHipmZvB6DhAAZLEHzvA41aXP2Pn/KTfWlg=="}' 2024-04-25T09:12:01.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 208322d1-3dfc-4908-8081-a6defff8b8e6 -i td/repeer-on-acting-back/0/new.json 2024-04-25T09:12:01.558 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:12:01.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/0/new.json 2024-04-25T09:12:01.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhHipmZvB6DhAAZLEHzvA41aXP2Pn/KTfWlg== --osd-uuid 208322d1-3dfc-4908-8081-a6defff8b8e6 2024-04-25T09:12:01.613 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:01.622+0000 7f5210d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:01.635 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:01.646+0000 7f5210d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:01.636 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:01.646+0000 7f5210d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:01.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:01.646+0000 7f5210d297c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:12:01.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:01.646+0000 7f5210d297c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:12:01.637 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:01.646+0000 7f5210d297c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:12:01.666 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:01.678+0000 7f5210d297c0 -1 bluestore(td/repeer-on-acting-back/0) _read_fsid unparsable uuid 2024-04-25T09:12:08.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/0/keyring 2024-04-25T09:12:08.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:12:08.040 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:12:08.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:12:08.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:12:08.619 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:12:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:12:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:12:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:12:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:12:08.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:12:08.656 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:08.666+0000 7f32c945b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:08.691 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:08.702+0000 7f32c945b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:08.693 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:08.702+0000 7f32c945b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:09.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:12:09.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:12:09.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:12:09.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:12:09.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:12:09.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:09.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:12:09.085 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:12:09.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:09.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:09.260 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:09.270+0000 7f32c945b7c0 -1 Falling back to public interface 2024-04-25T09:12:09.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:10.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:10.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:10.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:12:10.523 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:12:10.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:10.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:10.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:11.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:11.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:11.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:12:11.959 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:12:11.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:11.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:12.031 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:12.042+0000 7f32c945b7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:12:12.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:13.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:13.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:13.372 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:12:13.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:12:13.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:13.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:13.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:14.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:14.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:14.932 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:12:14.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:12:14.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:14.932 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:15.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:16.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:16.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:16.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:12:16.378 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:12:16.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:16.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:16.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:17.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:17.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:17.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:12:17.907 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:12:17.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:17.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:18.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:19.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:19.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:19.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:12:19.338 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:12:19.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:19.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:19.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:20.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:20.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:20.905 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:12:20.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:12:20.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:20.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:21.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:22.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:22.346 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:12:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:12:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:22.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:22.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:23.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:23.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:23.923 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:12:23.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:12:23.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:23.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:24.348 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:25.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:25.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:25.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:12:25.350 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:12:25.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:25.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:26.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:27.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:27.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:27.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:12:27.394 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:12:27.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:27.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:28.632 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:29.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:29.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:29.634 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:12:29.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:12:29.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:29.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:30.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:31.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:31.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:31.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:12:31.499 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:12:31.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:31.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:32.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:33.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:33.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:33.631 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:12:33.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:12:33.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:33.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:34.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:35.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:35.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:35.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:12:35.842 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:12:35.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:35.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:12:36.317 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/313469443,v1:127.0.0.1:6803/313469443] [v2:127.0.0.1:6804/313469443,v1:127.0.0.1:6805/313469443] exists,up 208322d1-3dfc-4908-8081-a6defff8b8e6 2024-04-25T09:12:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:12:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:12:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:12:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:12:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 1 2024-04-25T09:12:36.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/1 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/1' 2024-04-25T09:12:36.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/1/journal' 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:12:36.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:12:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/1 2024-04-25T09:12:36.325 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:12:36.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e031849-6877-4773-a66d-c42d374047d7 2024-04-25T09:12:36.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3e031849-6877-4773-a66d-c42d374047d7' 2024-04-25T09:12:36.325 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 3e031849-6877-4773-a66d-c42d374047d7 2024-04-25T09:12:36.326 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:12:36.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEHipmLq3PFRAAWndC4rnkJ1KYre2K9qmFiw== 2024-04-25T09:12:36.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEHipmLq3PFRAAWndC4rnkJ1KYre2K9qmFiw=="}' 2024-04-25T09:12:36.351 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e031849-6877-4773-a66d-c42d374047d7 -i td/repeer-on-acting-back/1/new.json 2024-04-25T09:12:36.881 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:12:36.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/1/new.json 2024-04-25T09:12:36.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEHipmLq3PFRAAWndC4rnkJ1KYre2K9qmFiw== --osd-uuid 3e031849-6877-4773-a66d-c42d374047d7 2024-04-25T09:12:36.942 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:36.957+0000 7f1d149c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:36.944 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:36.957+0000 7f1d149c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:36.945 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:36.957+0000 7f1d149c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:36.946 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:36.961+0000 7f1d149c67c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:12:36.946 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:36.961+0000 7f1d149c67c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:12:36.946 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:36.961+0000 7f1d149c67c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:12:36.990 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:37.005+0000 7f1d149c67c0 -1 bluestore(td/repeer-on-acting-back/1) _read_fsid unparsable uuid 2024-04-25T09:12:43.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/1/keyring 2024-04-25T09:12:43.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:12:43.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:12:43.755 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T09:12:43.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:12:44.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:12:44.321 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:12:44.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:12:44.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:12:44.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:12:44.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:12:44.359 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:44.373+0000 7f52399b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:44.391 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:44.405+0000 7f52399b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:44.392 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:44.405+0000 7f52399b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:44.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:12:44.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:12:44.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:12:44.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:12:44.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:12:44.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:44.749 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:12:44.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:12:44.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:44.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:44.939 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:44.953+0000 7f52399b97c0 -1 Falling back to public interface 2024-04-25T09:12:45.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:46.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:46.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:46.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:12:46.198 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:12:46.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:46.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:46.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:47.549 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:12:47.561+0000 7f52399b97c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:12:47.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:47.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:47.638 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:12:47.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:12:47.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:47.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:48.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:49.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:49.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:49.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:12:49.312 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:12:49.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:49.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:49.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:50.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:50.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:50.809 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:12:50.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:12:50.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:50.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:51.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:52.254 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:52.254 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:12:52.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:52.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:12:52.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:52.255 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:52.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:12:53.810 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:12:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:54.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:55.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:55.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:55.247 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:12:55.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:12:55.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:55.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:55.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:56.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:56.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:56.700 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:12:56.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:12:56.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:56.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:57.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:58.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:58.171 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:58.171 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:12:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:12:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:58.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:58.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:59.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:59.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:59.598 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:12:59.598 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:12:59.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:59.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:00.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:01.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:01.096 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:13:01.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:01.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:13:01.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:01.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:02.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:03.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:03.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:03.108 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:13:03.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:13:03.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:03.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:03.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:04.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:04.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:04.966 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:13:04.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:13:04.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:04.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:05.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:06.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:06.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:06.791 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:13:06.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:13:06.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:06.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:07.551 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:08.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:08.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:08.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:13:08.554 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:13:08.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:08.554 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:09.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:10.825 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:10.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:10.826 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:13:10.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:13:10.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:10.827 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:11.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:12.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:12.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:12.605 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:13:12.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:13:12.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:12.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:13:13.052 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1578512400,v1:127.0.0.1:6811/1578512400] [v2:127.0.0.1:6812/1578512400,v1:127.0.0.1:6813/1578512400] exists,up 3e031849-6877-4773-a66d-c42d374047d7 2024-04-25T09:13:13.055 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 2 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/2 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:13:13.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:13:13.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:13:13.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/2 2024-04-25T09:13:13.059 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:13:13.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8e873a7-85ae-40f0-b721-4f8917fe8844 2024-04-25T09:13:13.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b8e873a7-85ae-40f0-b721-4f8917fe8844' 2024-04-25T09:13:13.059 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 b8e873a7-85ae-40f0-b721-4f8917fe8844 2024-04-25T09:13:13.060 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:13:13.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpHipmmHnqBRAA8HgSq+EZEIr5esSfA4mnpQ== 2024-04-25T09:13:13.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpHipmmHnqBRAA8HgSq+EZEIr5esSfA4mnpQ=="}' 2024-04-25T09:13:13.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8e873a7-85ae-40f0-b721-4f8917fe8844 -i td/repeer-on-acting-back/2/new.json 2024-04-25T09:13:13.587 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:13:13.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/2/new.json 2024-04-25T09:13:13.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpHipmmHnqBRAA8HgSq+EZEIr5esSfA4mnpQ== --osd-uuid b8e873a7-85ae-40f0-b721-4f8917fe8844 2024-04-25T09:13:13.647 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:13.661+0000 7fdedafca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:13.704 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:13.717+0000 7fdedafca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:13.706 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:13.721+0000 7fdedafca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:13.706 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:13.721+0000 7fdedafca7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:13:13.706 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:13.721+0000 7fdedafca7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:13:13.706 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:13.721+0000 7fdedafca7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:13:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:13.845+0000 7fdedafca7c0 -1 bluestore(td/repeer-on-acting-back/2) _read_fsid unparsable uuid 2024-04-25T09:13:20.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/2/keyring 2024-04-25T09:13:20.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:13:20.217 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:13:20.218 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T09:13:20.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:13:20.829 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:13:20.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:13:20.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:13:20.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:13:20.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:13:20.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:13:20.867 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:20.881+0000 7fc26600c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:20.902 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:20.917+0000 7fc26600c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:20.904 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:20.917+0000 7fc26600c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:21.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:13:21.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:13:21.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:13:21.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:13:21.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:13:21.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:21.275 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:13:21.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:13:21.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:21.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:21.499 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:21.513+0000 7fc26600c7c0 -1 Falling back to public interface 2024-04-25T09:13:21.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:22.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:22.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:22.744 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:13:22.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:13:22.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:22.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:23.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:24.127 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:24.141+0000 7fc26600c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:13:24.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:24.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:24.184 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:13:24.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:13:24.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:24.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:24.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:25.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:25.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:25.915 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:13:25.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:13:25.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:25.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:26.550 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:27.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:27.552 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:27.552 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:13:27.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:13:27.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:27.553 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:28.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:29.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:29.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:29.233 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:13:29.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:13:29.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:29.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:29.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:30.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:30.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:30.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:13:30.684 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:13:30.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:30.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:31.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:32.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:32.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:32.121 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:13:32.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:13:32.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:32.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:32.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:33.588 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:33.589 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:13:33.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:33.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:13:33.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:33.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:34.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:35.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:35.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:35.234 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:13:35.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:13:35.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:35.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:35.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:36.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:36.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:36.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:13:36.715 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:13:36.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:36.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:37.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:38.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:38.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:38.245 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:13:38.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:13:38.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:38.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:39.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:40.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:40.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:40.198 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:13:40.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:13:40.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:40.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:41.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:42.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:42.316 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:13:42.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:13:42.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:42.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:43.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:44.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:44.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:44.262 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:13:44.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:13:44.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:44.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:45.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:46.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:46.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:46.637 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:13:46.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:13:46.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:46.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:47.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:48.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:48.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:48.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:13:48.435 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:13:48.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:48.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:49.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:13:50.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:13:50.477 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:50.477 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:13:50.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:13:50.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:50.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:13:50.906 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4185210474,v1:127.0.0.1:6819/4185210474] [v2:127.0.0.1:6820/4185210474,v1:127.0.0.1:6821/4185210474] exists,up b8e873a7-85ae-40f0-b721-4f8917fe8844 2024-04-25T09:13:50.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:13:50.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:13:50.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:13:50.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:13:50.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 3 2024-04-25T09:13:50.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/3 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/3' 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/3/journal' 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:13:50.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:13:50.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:13:50.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:13:50.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:13:50.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:13:50.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:13:50.911 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:13:50.911 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:13:50.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:13:50.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/3 2024-04-25T09:13:50.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:13:50.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ca48454-4ee4-4f46-841c-f9c67b9da990 2024-04-25T09:13:50.914 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2ca48454-4ee4-4f46-841c-f9c67b9da990' 2024-04-25T09:13:50.914 INFO:tasks.workunit.client.0.smithi088.stdout:add osd3 2ca48454-4ee4-4f46-841c-f9c67b9da990 2024-04-25T09:13:50.914 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:13:50.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOHipmOhEIORAA00e9lMTHULrdmkl7N6qAhQ== 2024-04-25T09:13:50.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOHipmOhEIORAA00e9lMTHULrdmkl7N6qAhQ=="}' 2024-04-25T09:13:50.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ca48454-4ee4-4f46-841c-f9c67b9da990 -i td/repeer-on-acting-back/3/new.json 2024-04-25T09:13:51.431 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:13:51.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/3/new.json 2024-04-25T09:13:51.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOHipmOhEIORAA00e9lMTHULrdmkl7N6qAhQ== --osd-uuid 2ca48454-4ee4-4f46-841c-f9c67b9da990 2024-04-25T09:13:51.487 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:51.500+0000 7f8f27df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:51.499 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:51.512+0000 7f8f27df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:51.501 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:51.512+0000 7f8f27df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:51.501 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:51.512+0000 7f8f27df87c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:13:51.501 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:51.512+0000 7f8f27df87c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:13:51.501 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:51.512+0000 7f8f27df87c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:13:51.540 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:51.552+0000 7f8f27df87c0 -1 bluestore(td/repeer-on-acting-back/3) _read_fsid unparsable uuid 2024-04-25T09:13:58.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/3/keyring 2024-04-25T09:13:58.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:13:58.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:13:58.228 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd3 key to auth repository 2024-04-25T09:13:58.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:13:58.808 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.3 2024-04-25T09:13:58.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:13:58.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:13:58.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:13:58.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:13:58.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:13:58.846 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:58.860+0000 7f0a3b54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:58.872 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:58.884+0000 7f0a3b54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:58.873 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:58.888+0000 7f0a3b54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:13:59.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:13:59.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:13:59.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:13:59.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:13:59.232 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:13:59.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:13:59.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:13:59.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:13:59.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:13:59.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:13:59.439 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:13:59.452+0000 7f0a3b54f7c0 -1 Falling back to public interface 2024-04-25T09:13:59.642 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:00.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:00.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:00.644 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:14:00.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:14:00.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:00.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:01.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:02.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:02.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:02.082 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:14:02.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:14:02.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:02.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:02.112 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:02.124+0000 7f0a3b54f7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:14:02.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:03.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:03.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:03.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:14:03.648 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:14:03.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:03.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:04.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:05.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:05.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:05.192 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:14:05.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:14:05.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:05.193 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:05.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:06.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:06.629 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:06.629 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:14:06.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:14:06.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:06.630 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:07.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:08.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:08.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:08.105 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:14:08.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:14:08.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:08.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:08.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:09.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:09.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:09.669 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:14:09.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:14:09.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:09.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:10.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:11.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:11.183 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:14:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:14:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:11.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:11.635 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:12.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:12.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:12.637 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:14:12.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:14:12.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:12.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:13.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:14.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:14.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:14.124 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:14:14.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:14:14.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:14.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:14.653 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:15.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:15.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:15.655 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:14:15.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:14:15.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:15.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:16.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:17.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:17.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:17.626 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:14:17.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:14:17.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:17.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:18.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:19.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:19.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:19.319 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:14:19.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:14:19.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:19.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:20.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:21.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:21.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:21.133 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:14:21.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:14:21.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:21.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:22.370 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:23.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:23.371 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:23.372 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:14:23.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:14:23.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:23.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:24.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:25.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:25.180 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:14:25.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:25.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:14:25.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:25.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:26.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:27.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:27.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:27.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:14:27.422 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:14:27.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:27.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:14:27.954 INFO:tasks.workunit.client.0.smithi088.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2777181439,v1:127.0.0.1:6827/2777181439] [v2:127.0.0.1:6828/2777181439,v1:127.0.0.1:6829/2777181439] exists,up 2ca48454-4ee4-4f46-841c-f9c67b9da990 2024-04-25T09:14:27.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:14:27.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 4 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/4 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:14:27.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/4' 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/4/journal' 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:14:27.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:14:27.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/4 2024-04-25T09:14:27.961 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:14:27.961 INFO:tasks.workunit.client.0.smithi088.stdout:add osd4 ee6f6e2d-c773-4958-8008-2f6d04f8b65e 2024-04-25T09:14:27.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee6f6e2d-c773-4958-8008-2f6d04f8b65e 2024-04-25T09:14:27.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 ee6f6e2d-c773-4958-8008-2f6d04f8b65e' 2024-04-25T09:14:27.962 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:14:27.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0HipmONsXABAA13o4TdUOicKAMpyhyD+cMA== 2024-04-25T09:14:27.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0HipmONsXABAA13o4TdUOicKAMpyhyD+cMA=="}' 2024-04-25T09:14:27.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee6f6e2d-c773-4958-8008-2f6d04f8b65e -i td/repeer-on-acting-back/4/new.json 2024-04-25T09:14:28.555 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:14:28.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/4/new.json 2024-04-25T09:14:28.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0HipmONsXABAA13o4TdUOicKAMpyhyD+cMA== --osd-uuid ee6f6e2d-c773-4958-8008-2f6d04f8b65e 2024-04-25T09:14:28.615 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:28.627+0000 7f76bb4037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:28.617 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:28.631+0000 7f76bb4037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:28.619 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:28.631+0000 7f76bb4037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:28.619 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:28.631+0000 7f76bb4037c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:14:28.619 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:28.631+0000 7f76bb4037c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:14:28.619 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:28.631+0000 7f76bb4037c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:14:28.660 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:28.671+0000 7f76bb4037c0 -1 bluestore(td/repeer-on-acting-back/4) _read_fsid unparsable uuid 2024-04-25T09:14:35.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/4/keyring 2024-04-25T09:14:35.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:14:35.391 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd4 key to auth repository 2024-04-25T09:14:35.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T09:14:35.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:14:35.969 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.4 2024-04-25T09:14:35.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T09:14:35.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:14:35.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:14:35.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:14:35.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:14:36.009 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:36.019+0000 7f9670bc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:36.018 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:36.031+0000 7f9670bc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:36.020 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:36.031+0000 7f9670bc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:36.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T09:14:36.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:14:36.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T09:14:36.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:14:36.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:14:36.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:36.407 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:14:36.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:14:36.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:36.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:36.603 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:36.615+0000 7f9670bc57c0 -1 Falling back to public interface 2024-04-25T09:14:36.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:37.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:37.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:37.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:14:37.873 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:14:37.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:37.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:38.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:39.190 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:14:39.203+0000 7f9670bc57c0 -1 osd.4 0 log_to_monitors true 2024-04-25T09:14:39.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:39.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:39.298 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:14:39.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:14:39.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:39.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:39.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:40.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:40.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:40.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:14:40.863 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:14:40.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:40.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:41.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:42.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:42.319 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:14:42.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:14:42.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:42.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:42.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:43.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:43.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:43.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:14:43.811 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:14:43.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:43.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:44.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:45.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:45.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:45.333 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:14:45.334 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:14:45.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:45.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:45.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:46.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:46.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:46.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:14:46.820 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:14:46.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:46.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:47.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:48.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:48.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:48.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:14:48.379 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:14:48.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:48.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:48.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:49.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:49.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:49.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:14:49.895 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:14:49.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:49.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:50.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:51.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:51.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:51.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:14:51.376 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:14:51.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:51.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:51.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:52.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:52.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:52.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:14:52.843 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:14:52.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:52.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:54.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:14:54.337 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:14:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:54.337 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:55.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:56.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:56.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:56.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:14:56.409 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:14:56.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:56.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:57.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:58.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:58.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:58.134 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:14:58.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:14:58.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:58.135 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:14:58.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:59.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:59.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:59.720 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:14:59.720 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:14:59.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:59.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:15:00.197 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:01.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:01.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:01.200 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:15:01.200 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:15:01.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:01.201 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:15:02.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:03.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:03.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:03.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:15:03.039 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:15:03.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:03.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:15:04.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:05.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:05.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:05.388 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:15:05.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:15:05.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:05.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:15:05.835 INFO:tasks.workunit.client.0.smithi088.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2993883471,v1:127.0.0.1:6835/2993883471] [v2:127.0.0.1:6836/2993883471,v1:127.0.0.1:6837/2993883471] exists,up ee6f6e2d-c773-4958-8008-2f6d04f8b65e 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 5 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T09:15:05.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/5 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/5' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/5/journal' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:15:05.840 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:15:05.841 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:15:05.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/5 2024-04-25T09:15:05.842 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:15:05.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=afc1d8fd-424a-4ae9-8651-4904e0cb9453 2024-04-25T09:15:05.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 afc1d8fd-424a-4ae9-8651-4904e0cb9453' 2024-04-25T09:15:05.843 INFO:tasks.workunit.client.0.smithi088.stdout:add osd5 afc1d8fd-424a-4ae9-8651-4904e0cb9453 2024-04-25T09:15:05.843 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:15:05.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZHypmIlexNBAApV6J7DXQQZ2vEzUg/QRSAw== 2024-04-25T09:15:05.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZHypmIlexNBAApV6J7DXQQZ2vEzUg/QRSAw=="}' 2024-04-25T09:15:05.868 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new afc1d8fd-424a-4ae9-8651-4904e0cb9453 -i td/repeer-on-acting-back/5/new.json 2024-04-25T09:15:06.357 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:15:06.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/5/new.json 2024-04-25T09:15:06.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZHypmIlexNBAApV6J7DXQQZ2vEzUg/QRSAw== --osd-uuid afc1d8fd-424a-4ae9-8651-4904e0cb9453 2024-04-25T09:15:06.413 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:06.427+0000 7f2bef8ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:06.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:06.443+0000 7f2bef8ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:06.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:06.443+0000 7f2bef8ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:06.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:06.447+0000 7f2bef8ee7c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:15:06.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:06.447+0000 7f2bef8ee7c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:15:06.435 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:06.447+0000 7f2bef8ee7c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:15:06.562 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:06.575+0000 7f2bef8ee7c0 -1 bluestore(td/repeer-on-acting-back/5) _read_fsid unparsable uuid 2024-04-25T09:15:13.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/5/keyring 2024-04-25T09:15:13.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:15:13.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T09:15:13.341 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd5 key to auth repository 2024-04-25T09:15:13.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:15:13.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T09:15:13.938 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.5 2024-04-25T09:15:13.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:15:13.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:15:13.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:15:13.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:15:13.975 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:13.986+0000 7f09397b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:13.987 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:13.998+0000 7f09397b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:13.989 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:14.002+0000 7f09397b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:14.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T09:15:14.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:15:14.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T09:15:14.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:15:14.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:15:14.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:14.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:15:14.374 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:15:14.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:14.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:14.571 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:14.582+0000 7f09397b97c0 -1 Falling back to public interface 2024-04-25T09:15:14.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:15.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:15.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:15.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:15:15.795 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:15:15.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:15.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:16.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:17.162 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:15:17.174+0000 7f09397b97c0 -1 osd.5 0 log_to_monitors true 2024-04-25T09:15:17.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:17.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:17.210 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:15:17.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:15:17.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:17.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:17.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:19.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:19.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:19.002 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:15:19.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:15:19.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:19.003 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:19.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:20.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:20.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:20.438 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:15:20.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:15:20.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:20.438 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:21.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:22.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:22.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:22.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:15:22.063 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:15:22.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:22.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:22.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:23.622 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:23.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:23.623 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:15:23.623 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:15:23.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:23.624 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:24.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:25.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:25.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:25.325 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:15:25.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:15:25.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:25.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:25.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:26.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:26.780 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:15:26.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:26.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:15:26.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:26.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:27.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:28.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:28.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:28.233 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:15:28.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:15:28.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:28.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:28.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:29.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:29.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:29.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:15:29.783 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:15:29.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:29.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:30.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:31.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:31.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:31.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:15:31.601 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:15:31.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:31.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:32.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:33.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:33.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:33.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:15:33.515 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:15:33.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:33.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:34.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:35.327 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:35.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:35.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:15:35.328 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:15:35.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:35.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:36.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:37.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:37.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:37.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:15:37.133 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:15:37.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:37.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:38.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:39.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:39.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:39.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:15:39.101 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:15:39.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:39.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:39.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:40.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:40.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:40.975 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:15:40.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:15:40.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:40.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:41.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:42.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:42.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:42.562 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:15:42.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:15:42.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:42.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:15:43.004 INFO:tasks.workunit.client.0.smithi088.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3306304008,v1:127.0.0.1:6843/3306304008] [v2:127.0.0.1:6844/3306304008,v1:127.0.0.1:6845/3306304008] exists,up afc1d8fd-424a-4ae9-8651-4904e0cb9453 2024-04-25T09:15:43.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:15:43.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:15:43.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:15:43.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:60: TEST_repeer_on_down_acting_member_coming_back: create_pool test 1 1 2024-04-25T09:15:43.005 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T09:15:43.735 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T09:15:43.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:15:44.756 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:61: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test size 3 2024-04-25T09:15:45.505 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 size to 3 2024-04-25T09:15:45.529 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:62: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test min_size 2 2024-04-25T09:15:46.335 INFO:tasks.workunit.client.0.smithi088.stderr:set pool 1 min_size to 2 2024-04-25T09:15:46.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: ceph pg dump pools -f json 2024-04-25T09:15:46.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq .pool_stats 2024-04-25T09:15:46.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq '.[].poolid' 2024-04-25T09:15:46.769 INFO:tasks.workunit.client.0.smithi088.stderr:dumped pools 2024-04-25T09:15:46.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: local poolid=1 2024-04-25T09:15:46.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:64: TEST_repeer_on_down_acting_member_coming_back: local pgid=1.0 2024-04-25T09:15:46.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:67: TEST_repeer_on_down_acting_member_coming_back: ceph osd set-require-min-compat-client luminous 2024-04-25T09:15:47.515 INFO:tasks.workunit.client.0.smithi088.stderr:set require_min_compat_client to luminous 2024-04-25T09:15:47.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:69: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 2 3 2024-04-25T09:15:48.520 INFO:tasks.workunit.client.0.smithi088.stderr:set 1.0 pg_upmap mapping to [1,2,3] 2024-04-25T09:15:48.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:71: TEST_repeer_on_down_acting_member_coming_back: flush_pg_stats 2024-04-25T09:15:48.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:15:48.544 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:15:49.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:15:49.107 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:15:49.107 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T09:15:49.107 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T09:15:49.107 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T09:15:49.107 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T09:15:49.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:15:49.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:49.108 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:15:49.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836527 2024-04-25T09:15:49.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836527 2024-04-25T09:15:49.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527' 2024-04-25T09:15:49.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:49.322 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:15:49.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672998 2024-04-25T09:15:49.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672998 2024-04-25T09:15:49.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672998' 2024-04-25T09:15:49.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:49.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:15:49.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509471 2024-04-25T09:15:49.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509471 2024-04-25T09:15:49.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672998 2-64424509471' 2024-04-25T09:15:49.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:49.652 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:15:49.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345944 2024-04-25T09:15:49.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345944 2024-04-25T09:15:49.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672998 2-64424509471 3-85899345944' 2024-04-25T09:15:49.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:49.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:15:49.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182418 2024-04-25T09:15:49.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182418 2024-04-25T09:15:49.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672998 2-64424509471 3-85899345944 4-107374182418' 2024-04-25T09:15:49.976 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:49.976 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:15:50.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051593 2024-04-25T09:15:50.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051593 2024-04-25T09:15:50.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672998 2-64424509471 3-85899345944 4-107374182418 5-124554051593' 2024-04-25T09:15:50.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:50.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836527 2024-04-25T09:15:50.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:15:50.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836527 2024-04-25T09:15:50.146 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:50.146 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836527 2024-04-25T09:15:50.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836527 2024-04-25T09:15:50.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-04-25T09:15:50.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:15:50.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836527 2024-04-25T09:15:50.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:50.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:50.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672998 2024-04-25T09:15:50.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:15:50.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672998 2024-04-25T09:15:50.572 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:50.573 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949672998 2024-04-25T09:15:50.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672998 2024-04-25T09:15:50.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-04-25T09:15:50.574 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:15:50.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949672998 2024-04-25T09:15:50.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:50.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509471 2024-04-25T09:15:50.993 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:50.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:15:50.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509471 2024-04-25T09:15:50.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:50.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509471 2024-04-25T09:15:50.997 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509471 2024-04-25T09:15:50.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-04-25T09:15:50.997 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:15:51.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509471 -lt 64424509471 2024-04-25T09:15:51.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:51.449 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345944 2024-04-25T09:15:51.450 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:51.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:15:51.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345944 2024-04-25T09:15:51.456 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:51.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345944 2024-04-25T09:15:51.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345944' 2024-04-25T09:15:51.456 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345944 2024-04-25T09:15:51.457 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:15:51.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345944 -lt 85899345944 2024-04-25T09:15:51.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:51.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:51.898 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182418 2024-04-25T09:15:51.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:15:51.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182418 2024-04-25T09:15:51.901 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:51.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182418 2024-04-25T09:15:51.903 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182418 2024-04-25T09:15:51.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182418' 2024-04-25T09:15:51.903 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:15:52.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182419 -lt 107374182418 2024-04-25T09:15:52.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:52.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-124554051593 2024-04-25T09:15:52.364 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:52.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:15:52.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-124554051593 2024-04-25T09:15:52.367 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:52.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051593 2024-04-25T09:15:52.369 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 124554051593 2024-04-25T09:15:52.369 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 124554051593' 2024-04-25T09:15:52.369 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:15:52.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051593 -lt 124554051593 2024-04-25T09:15:52.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:72: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:15:52.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:15:52.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:15:52.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:15:52.830 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:15:52.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:15:52.831 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:15:52.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:15:52.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:15:52.831 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:15:52.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:15:52.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:15:52.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:15:52.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:15:52.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:15:52.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:53.357 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:15:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836530 2024-04-25T09:15:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836530 2024-04-25T09:15:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530' 2024-04-25T09:15:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:53.540 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:15:53.692 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673001 2024-04-25T09:15:53.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673001 2024-04-25T09:15:53.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673001' 2024-04-25T09:15:53.693 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:53.693 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:15:53.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509474 2024-04-25T09:15:53.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509474 2024-04-25T09:15:53.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673001 2-64424509474' 2024-04-25T09:15:53.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:53.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:15:54.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345947 2024-04-25T09:15:54.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345947 2024-04-25T09:15:54.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673001 2-64424509474 3-85899345947' 2024-04-25T09:15:54.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:54.026 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:15:54.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182421 2024-04-25T09:15:54.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182421 2024-04-25T09:15:54.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673001 2-64424509474 3-85899345947 4-107374182421' 2024-04-25T09:15:54.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:15:54.185 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:15:54.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051596 2024-04-25T09:15:54.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051596 2024-04-25T09:15:54.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673001 2-64424509474 3-85899345947 4-107374182421 5-124554051596' 2024-04-25T09:15:54.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:54.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836530 2024-04-25T09:15:54.376 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:54.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:15:54.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836530 2024-04-25T09:15:54.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:54.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836530 2024-04-25T09:15:54.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-04-25T09:15:54.379 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836530 2024-04-25T09:15:54.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:15:54.777 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836530 -lt 21474836530 2024-04-25T09:15:54.778 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:54.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673001 2024-04-25T09:15:54.778 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:54.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:15:54.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673001 2024-04-25T09:15:54.780 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:54.782 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673001 2024-04-25T09:15:54.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673001 2024-04-25T09:15:54.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-04-25T09:15:54.782 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:15:55.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673001 -lt 42949673001 2024-04-25T09:15:55.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:55.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:55.217 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509474 2024-04-25T09:15:55.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:15:55.219 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:55.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509474 2024-04-25T09:15:55.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509474 2024-04-25T09:15:55.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-04-25T09:15:55.221 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509474 2024-04-25T09:15:55.221 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:15:55.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509472 -lt 64424509474 2024-04-25T09:15:55.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:15:56.701 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:15:56.702 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:15:57.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509475 -lt 64424509474 2024-04-25T09:15:57.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:57.116 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345947 2024-04-25T09:15:57.116 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:57.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:15:57.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345947 2024-04-25T09:15:57.119 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:57.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345947 2024-04-25T09:15:57.120 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345947 2024-04-25T09:15:57.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345947' 2024-04-25T09:15:57.121 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:15:57.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345947 -lt 85899345947 2024-04-25T09:15:57.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:57.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182421 2024-04-25T09:15:57.564 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:57.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:15:57.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182421 2024-04-25T09:15:57.566 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:57.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182421 2024-04-25T09:15:57.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182421' 2024-04-25T09:15:57.568 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182421 2024-04-25T09:15:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:15:58.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182422 -lt 107374182421 2024-04-25T09:15:58.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:15:58.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:15:58.010 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-124554051596 2024-04-25T09:15:58.011 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:15:58.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-124554051596 2024-04-25T09:15:58.012 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:15:58.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051596 2024-04-25T09:15:58.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 124554051596' 2024-04-25T09:15:58.014 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 124554051596 2024-04-25T09:15:58.014 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:15:58.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051596 -lt 124554051596 2024-04-25T09:15:58.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:15:58.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:15:58.482 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:15:59.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:15:59.024 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:15:59.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:15:59.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:15:59.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:15:59.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:15:59.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:15:59.025 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:15:59.440 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:15:59.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:15:59.441 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:15:59.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:15:59.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:15:59.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:15:59.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:15:59.968 INFO:tasks.workunit.client.0.smithi088.stdout:writing initial objects 2024-04-25T09:15:59.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:74: TEST_repeer_on_down_acting_member_coming_back: echo 'writing initial objects' 2024-04-25T09:15:59.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: seq 1 100 2024-04-25T09:15:59.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:15:59.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_1 /etc/fstab 2024-04-25T09:16:00.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:00.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_2 /etc/fstab 2024-04-25T09:16:00.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:00.335 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_3 /etc/fstab 2024-04-25T09:16:00.544 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:00.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_4 /etc/fstab 2024-04-25T09:16:00.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:00.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_5 /etc/fstab 2024-04-25T09:16:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:00.921 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_6 /etc/fstab 2024-04-25T09:16:01.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:01.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_7 /etc/fstab 2024-04-25T09:16:01.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:01.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_8 /etc/fstab 2024-04-25T09:16:01.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:01.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_9 /etc/fstab 2024-04-25T09:16:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:01.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_10 /etc/fstab 2024-04-25T09:16:01.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:01.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_11 /etc/fstab 2024-04-25T09:16:01.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:01.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_12 /etc/fstab 2024-04-25T09:16:02.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:02.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_13 /etc/fstab 2024-04-25T09:16:02.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:02.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_14 /etc/fstab 2024-04-25T09:16:02.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:02.549 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_15 /etc/fstab 2024-04-25T09:16:02.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:02.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_16 /etc/fstab 2024-04-25T09:16:02.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:02.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_17 /etc/fstab 2024-04-25T09:16:03.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:03.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_18 /etc/fstab 2024-04-25T09:16:03.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:03.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_19 /etc/fstab 2024-04-25T09:16:03.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:03.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_20 /etc/fstab 2024-04-25T09:16:03.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:03.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_21 /etc/fstab 2024-04-25T09:16:03.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:03.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_22 /etc/fstab 2024-04-25T09:16:03.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:03.913 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_23 /etc/fstab 2024-04-25T09:16:04.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:04.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_24 /etc/fstab 2024-04-25T09:16:04.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:04.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_25 /etc/fstab 2024-04-25T09:16:04.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:04.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_26 /etc/fstab 2024-04-25T09:16:04.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:04.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_27 /etc/fstab 2024-04-25T09:16:04.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:04.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_28 /etc/fstab 2024-04-25T09:16:05.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:05.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_29 /etc/fstab 2024-04-25T09:16:05.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:05.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_30 /etc/fstab 2024-04-25T09:16:05.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:05.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_31 /etc/fstab 2024-04-25T09:16:05.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:05.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_32 /etc/fstab 2024-04-25T09:16:05.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:05.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_33 /etc/fstab 2024-04-25T09:16:05.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:05.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_34 /etc/fstab 2024-04-25T09:16:06.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:06.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_35 /etc/fstab 2024-04-25T09:16:06.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:06.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_36 /etc/fstab 2024-04-25T09:16:06.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:06.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_37 /etc/fstab 2024-04-25T09:16:06.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:06.585 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_38 /etc/fstab 2024-04-25T09:16:06.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:06.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_39 /etc/fstab 2024-04-25T09:16:06.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:06.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_40 /etc/fstab 2024-04-25T09:16:07.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:07.104 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_41 /etc/fstab 2024-04-25T09:16:07.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:07.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_42 /etc/fstab 2024-04-25T09:16:07.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:07.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_43 /etc/fstab 2024-04-25T09:16:07.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:07.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_44 /etc/fstab 2024-04-25T09:16:07.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:07.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_45 /etc/fstab 2024-04-25T09:16:07.991 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:07.992 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_46 /etc/fstab 2024-04-25T09:16:08.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:08.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_47 /etc/fstab 2024-04-25T09:16:08.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:08.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_48 /etc/fstab 2024-04-25T09:16:08.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:08.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_49 /etc/fstab 2024-04-25T09:16:08.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:08.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_50 /etc/fstab 2024-04-25T09:16:08.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:08.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_51 /etc/fstab 2024-04-25T09:16:09.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:09.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_52 /etc/fstab 2024-04-25T09:16:09.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:09.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_53 /etc/fstab 2024-04-25T09:16:09.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:09.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_54 /etc/fstab 2024-04-25T09:16:09.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:09.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_55 /etc/fstab 2024-04-25T09:16:09.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:09.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_56 /etc/fstab 2024-04-25T09:16:09.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:09.903 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_57 /etc/fstab 2024-04-25T09:16:10.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:10.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_58 /etc/fstab 2024-04-25T09:16:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:10.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_59 /etc/fstab 2024-04-25T09:16:10.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:10.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_60 /etc/fstab 2024-04-25T09:16:10.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:10.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_61 /etc/fstab 2024-04-25T09:16:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_62 /etc/fstab 2024-04-25T09:16:10.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:10.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_63 /etc/fstab 2024-04-25T09:16:11.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:11.127 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_64 /etc/fstab 2024-04-25T09:16:11.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:11.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_65 /etc/fstab 2024-04-25T09:16:11.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:11.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_66 /etc/fstab 2024-04-25T09:16:11.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:11.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_67 /etc/fstab 2024-04-25T09:16:11.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:11.819 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_68 /etc/fstab 2024-04-25T09:16:11.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:11.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_69 /etc/fstab 2024-04-25T09:16:12.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:12.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_70 /etc/fstab 2024-04-25T09:16:12.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:12.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_71 /etc/fstab 2024-04-25T09:16:12.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:12.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_72 /etc/fstab 2024-04-25T09:16:12.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:12.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_73 /etc/fstab 2024-04-25T09:16:12.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:12.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_74 /etc/fstab 2024-04-25T09:16:13.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:13.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_75 /etc/fstab 2024-04-25T09:16:13.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:13.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_76 /etc/fstab 2024-04-25T09:16:13.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:13.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_77 /etc/fstab 2024-04-25T09:16:13.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:13.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_78 /etc/fstab 2024-04-25T09:16:13.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:13.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_79 /etc/fstab 2024-04-25T09:16:14.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:14.015 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_80 /etc/fstab 2024-04-25T09:16:14.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:14.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_81 /etc/fstab 2024-04-25T09:16:14.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:14.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_82 /etc/fstab 2024-04-25T09:16:14.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:14.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_83 /etc/fstab 2024-04-25T09:16:14.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:14.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_84 /etc/fstab 2024-04-25T09:16:14.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:14.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_85 /etc/fstab 2024-04-25T09:16:15.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:15.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_86 /etc/fstab 2024-04-25T09:16:15.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:15.306 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_87 /etc/fstab 2024-04-25T09:16:15.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:15.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_88 /etc/fstab 2024-04-25T09:16:15.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:15.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_89 /etc/fstab 2024-04-25T09:16:15.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:15.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_90 /etc/fstab 2024-04-25T09:16:16.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:16.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_91 /etc/fstab 2024-04-25T09:16:16.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:16.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_92 /etc/fstab 2024-04-25T09:16:16.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:16.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_93 /etc/fstab 2024-04-25T09:16:16.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:16.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_94 /etc/fstab 2024-04-25T09:16:16.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:16.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_95 /etc/fstab 2024-04-25T09:16:16.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:16.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_96 /etc/fstab 2024-04-25T09:16:17.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:17.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_97 /etc/fstab 2024-04-25T09:16:17.321 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:17.322 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_98 /etc/fstab 2024-04-25T09:16:17.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:17.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_99 /etc/fstab 2024-04-25T09:16:17.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:16:17.902 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_100 /etc/fstab 2024-04-25T09:16:18.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T09:16:18.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:16:18.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:16:18.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:16:18.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:16:18.078 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T09:16:18.079 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:16:18.079 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:16:18.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:16:18.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:16:18.080 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:16:18.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T09:16:18.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:16:18.154 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:16:18.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:16:18.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:16:18.155 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:16:18.590 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:16:18.591 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:16:18.591 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T09:16:18.591 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T09:16:18.591 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T09:16:18.591 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T09:16:18.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:16:18.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:18.591 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:16:18.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836537 2024-04-25T09:16:18.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836537 2024-04-25T09:16:18.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537' 2024-04-25T09:16:18.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:18.759 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:16:18.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673008 2024-04-25T09:16:18.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673008 2024-04-25T09:16:18.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673008' 2024-04-25T09:16:18.919 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:18.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:16:19.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509481 2024-04-25T09:16:19.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509481 2024-04-25T09:16:19.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673008 2-64424509481' 2024-04-25T09:16:19.082 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:19.082 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:16:19.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345954 2024-04-25T09:16:19.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345954 2024-04-25T09:16:19.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673008 2-64424509481 3-85899345954' 2024-04-25T09:16:19.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:19.241 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:16:19.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182428 2024-04-25T09:16:19.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182428 2024-04-25T09:16:19.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673008 2-64424509481 3-85899345954 4-107374182428' 2024-04-25T09:16:19.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:19.398 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:16:19.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051603 2024-04-25T09:16:19.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051603 2024-04-25T09:16:19.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673008 2-64424509481 3-85899345954 4-107374182428 5-124554051603' 2024-04-25T09:16:19.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:19.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836537 2024-04-25T09:16:19.558 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:19.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:16:19.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:19.559 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836537 2024-04-25T09:16:19.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836537 2024-04-25T09:16:19.561 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836537 2024-04-25T09:16:19.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-04-25T09:16:19.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:16:19.962 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836537 -lt 21474836537 2024-04-25T09:16:19.963 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:19.963 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673008 2024-04-25T09:16:19.964 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:19.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:16:19.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673008 2024-04-25T09:16:19.966 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:19.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673008 2024-04-25T09:16:19.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-04-25T09:16:19.968 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673008 2024-04-25T09:16:19.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:16:20.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673008 -lt 42949673008 2024-04-25T09:16:20.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:20.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509481 2024-04-25T09:16:20.440 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:20.442 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:16:20.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509481 2024-04-25T09:16:20.442 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:20.443 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509481 2024-04-25T09:16:20.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-04-25T09:16:20.444 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509481 2024-04-25T09:16:20.444 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:16:20.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509481 -lt 64424509481 2024-04-25T09:16:20.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:20.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345954 2024-04-25T09:16:20.875 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:20.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:16:20.878 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345954 2024-04-25T09:16:20.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:20.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345954 2024-04-25T09:16:20.881 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345954 2024-04-25T09:16:20.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345954' 2024-04-25T09:16:20.881 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:16:21.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345954 -lt 85899345954 2024-04-25T09:16:21.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:21.310 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:21.311 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182428 2024-04-25T09:16:21.311 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:16:21.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182428 2024-04-25T09:16:21.312 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:21.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182428 2024-04-25T09:16:21.313 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182428 2024-04-25T09:16:21.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182428' 2024-04-25T09:16:21.314 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:16:21.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182428 -lt 107374182428 2024-04-25T09:16:21.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:21.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-124554051603 2024-04-25T09:16:21.741 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:21.742 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:16:21.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-124554051603 2024-04-25T09:16:21.743 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:21.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051603 2024-04-25T09:16:21.744 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 124554051603 2024-04-25T09:16:21.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 124554051603' 2024-04-25T09:16:21.745 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:16:22.203 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051603 -lt 124554051603 2024-04-25T09:16:22.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:16:22.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:22.204 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:22.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:16:22.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:16:22.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:16:22.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:16:22.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:16:22.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:16:22.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:16:22.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:16:23.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:16:23.245 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:16:23.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:23.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:23.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:16:23.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:16:23.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:16:23.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:84: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 4 5 2024-04-25T09:16:24.633 INFO:tasks.workunit.client.0.smithi088.stderr:set 1.0 pg_upmap mapping to [1,4,5] 2024-04-25T09:16:24.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:87: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:16:26.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:16:26.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:16:26.658 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:16:26.908 INFO:tasks.workunit.client.0.smithi088.stdout: 2, 2024-04-25T09:16:26.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:93: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back KILL osd.2 2024-04-25T09:16:26.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:16:26.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:16:26.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:16:26.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:16:26.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:16:27.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:16:27.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:94: TEST_repeer_on_down_acting_member_coming_back: ceph osd down osd.2 2024-04-25T09:16:27.623 INFO:tasks.workunit.client.0.smithi088.stderr:osd.2 is already down. 2024-04-25T09:16:27.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:97: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:16:29.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:16:29.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:16:29.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:16:29.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:103: TEST_repeer_on_down_acting_member_coming_back: activate_osd td/repeer-on-acting-back 2 2024-04-25T09:16:29.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:16:29.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:16:29.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repeer-on-acting-back/2 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:16:29.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:16:29.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:16:29.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:16:29.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repeer-on-acting-back/2 2024-04-25T09:16:29.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T09:16:29.834 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:16:29.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d851931a-f753-4bf1-93cc-180e651756ab --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:16:29.835 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repeer-on-acting-back/2/whoami 2024-04-25T09:16:29.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T09:16:29.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:16:29.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:16:29.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:16:29.865 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:29.877+0000 7f8a7a5717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:29.884 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:29.897+0000 7f8a7a5717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:29.885 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:29.901+0000 7f8a7a5717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:30.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T09:16:30.278 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:16:30.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:16:30.279 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:16:30.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:16:30.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:16:30.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:30.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:16:30.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:30.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:30.454 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:30.469+0000 7f8a7a5717c0 -1 Falling back to public interface 2024-04-25T09:16:30.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:31.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:31.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:31.738 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:16:31.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:16:31.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:31.739 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:32.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:33.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:33.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:33.168 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:16:33.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:16:33.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:33.169 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:33.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:34.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:34.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:34.607 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:16:34.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:16:34.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:34.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:34.714 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:34.729+0000 7f8a7a5717c0 -1 osd.2 46 log_to_monitors true 2024-04-25T09:16:35.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:36.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:36.183 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:16:36.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:36.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:16:36.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:36.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:36.819 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/3899353573,v1:127.0.0.1:6819/3899353573] [v2:127.0.0.1:6820/3899353573,v1:127.0.0.1:6821/3899353573] exists,up b8e873a7-85ae-40f0-b721-4f8917fe8844 2024-04-25T09:16:36.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:16:36.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:16:36.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:16:36.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:104: TEST_repeer_on_down_acting_member_coming_back: wait_for_osd up 2 2024-04-25T09:16:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:16:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:16:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:16:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:16:36.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:36.821 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:16:36.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:16:36.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:36.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:37.281 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/3899353573,v1:127.0.0.1:6819/3899353573] [v2:127.0.0.1:6820/3899353573,v1:127.0.0.1:6821/3899353573] exists,up b8e873a7-85ae-40f0-b721-4f8917fe8844 2024-04-25T09:16:37.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:16:37.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:16:37.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:16:37.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:107: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:16:39.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:16:39.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:16:39.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:16:39.463 INFO:tasks.workunit.client.0.smithi088.stdout: 2 2024-04-25T09:16:39.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T09:16:39.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:16:39.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:16:39.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:16:39.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:16:39.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T09:16:39.465 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:16:39.465 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:16:39.465 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:16:39.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:16:39.466 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:16:39.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T09:16:39.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:16:39.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:16:39.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:16:39.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:16:39.522 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr:2 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr:3 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr:4 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr:5' 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:39.995 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:16:40.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836543 2024-04-25T09:16:40.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836543 2024-04-25T09:16:40.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836543' 2024-04-25T09:16:40.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:40.229 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:16:40.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673014 2024-04-25T09:16:40.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673014 2024-04-25T09:16:40.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836543 1-42949673014' 2024-04-25T09:16:40.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:40.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:16:40.659 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364804 2024-04-25T09:16:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364804 2024-04-25T09:16:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836543 1-42949673014 2-214748364804' 2024-04-25T09:16:40.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:40.660 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:16:40.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345960 2024-04-25T09:16:40.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345960 2024-04-25T09:16:40.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836543 1-42949673014 2-214748364804 3-85899345960' 2024-04-25T09:16:40.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:40.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:16:40.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182435 2024-04-25T09:16:40.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182435 2024-04-25T09:16:40.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836543 1-42949673014 2-214748364804 3-85899345960 4-107374182435' 2024-04-25T09:16:40.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:16:40.972 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:16:41.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051609 2024-04-25T09:16:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051609 2024-04-25T09:16:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836543 1-42949673014 2-214748364804 3-85899345960 4-107374182435 5-124554051609' 2024-04-25T09:16:41.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:41.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836543 2024-04-25T09:16:41.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:41.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:16:41.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836543 2024-04-25T09:16:41.133 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836543 2024-04-25T09:16:41.134 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836543 2024-04-25T09:16:41.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-04-25T09:16:41.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:16:41.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836541 -lt 21474836543 2024-04-25T09:16:41.579 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:16:42.580 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:16:42.581 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:16:42.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836544 -lt 21474836543 2024-04-25T09:16:42.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:42.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673014 2024-04-25T09:16:42.999 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:43.000 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:16:43.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673014 2024-04-25T09:16:43.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:43.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673014 2024-04-25T09:16:43.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673014' 2024-04-25T09:16:43.002 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 42949673014 2024-04-25T09:16:43.002 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:16:43.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673014 -lt 42949673014 2024-04-25T09:16:43.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:43.484 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-214748364804 2024-04-25T09:16:43.485 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:43.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:16:43.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-214748364804 2024-04-25T09:16:43.487 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:43.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364804 2024-04-25T09:16:43.488 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 214748364804 2024-04-25T09:16:43.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 214748364804' 2024-04-25T09:16:43.489 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:16:43.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364804 -lt 214748364804 2024-04-25T09:16:43.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:43.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345960 2024-04-25T09:16:43.916 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:43.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:16:43.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345960 2024-04-25T09:16:43.919 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:43.919 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.3 seq 85899345960 2024-04-25T09:16:43.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345960 2024-04-25T09:16:43.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345960' 2024-04-25T09:16:43.920 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:16:44.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345961 -lt 85899345960 2024-04-25T09:16:44.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:44.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182435 2024-04-25T09:16:44.511 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:44.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:16:44.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182435 2024-04-25T09:16:44.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:44.514 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182435 2024-04-25T09:16:44.514 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.4 seq 107374182435 2024-04-25T09:16:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182435' 2024-04-25T09:16:44.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:16:44.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182435 -lt 107374182435 2024-04-25T09:16:44.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:16:44.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-124554051609 2024-04-25T09:16:44.946 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:16:44.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:16:44.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-124554051609 2024-04-25T09:16:44.948 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:16:44.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051609 2024-04-25T09:16:44.950 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.5 seq 124554051609 2024-04-25T09:16:44.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 124554051609' 2024-04-25T09:16:44.950 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:16:45.404 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051610 -lt 124554051609 2024-04-25T09:16:45.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:16:45.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:45.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:46.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:16:46.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:16:46.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:16:46.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:16:46.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:16:46.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:16:46.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:16:46.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:16:46.508 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:16:46.508 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:16:46.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:46.509 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:47.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:16:47.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:16:47.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:16:47.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:16:47.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:16:47.086 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:16:47.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:16:47.186 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:16:47.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:16:47.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:16:47.187 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:16:47.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:16:47.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:16:47.188 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:16:47.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:16:47.596 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:16:47.597 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:47.597 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:48.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:16:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:16:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:16:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:16:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:16:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:16:48.170 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:16:48.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:16:48.171 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:16:48.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2161 2024-04-25T09:16:48.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2161 '!=' null 2024-04-25T09:16:48.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:16:48.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:16:48.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:16:48.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:16:48.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:16:48.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:16:48.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:16:48.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:16:48.937 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:16:48.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:16:48.938 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:16:49.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:16:49.352 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:16:49.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:49.353 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:49.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:16:49.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:16:49.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:16:49.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:16:49.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:16:49.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:16:49.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:16:49.909 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:16:49.910 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:16:50.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2625 2024-04-25T09:16:50.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2625 '!=' null 2024-04-25T09:16:50.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:16:50.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:16:50.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:16:50.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:16:50.535 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:16:50.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:16:50.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:16:50.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:16:50.536 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:16:50.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:16:50.537 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:16:50.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:16:50.935 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:16:50.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:16:50.936 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:16:51.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:16:51.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:16:51.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:16:51.473 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: find td/repeer-on-acting-back -name '*osd*log' 2024-04-25T09:16:51.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: grep -q 'Active: got notify from previous acting member.*, requesting pg_temp change' td/repeer-on-acting-back/osd.5.log td/repeer-on-acting-back/osd.3.log td/repeer-on-acting-back/osd.4.log td/repeer-on-acting-back/osd.2.log td/repeer-on-acting-back/osd.1.log td/repeer-on-acting-back/osd.0.log 2024-04-25T09:16:51.486 INFO:tasks.workunit.client.0.smithi088.stdout:success 2024-04-25T09:16:51.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:119: TEST_repeer_on_down_acting_member_coming_back: echo success 2024-04-25T09:16:51.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:121: TEST_repeer_on_down_acting_member_coming_back: delete_pool test 2024-04-25T09:16:51.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T09:16:51.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T09:16:52.085 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' does not exist 2024-04-25T09:16:52.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:122: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back 2024-04-25T09:16:52.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:16:52.102 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:16:52.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:16:52.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:16:52.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:16:57.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:16:57.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:42: run: teardown td/repeer-on-acting-back 2024-04-25T09:16:57.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:16:57.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:16:57.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:16:57.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:16:57.422 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:16:57.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:16:57.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:16:57.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:16:57.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:16:57.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:16:57.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:16:57.432 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:16:57.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:16:57.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:16:57.433 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:16:57.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:57.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:16:57.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:16:57.434 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:57.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:16:57.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:16:57.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:16:57.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:16:57.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:57.477 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.478073 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/repeer-on-acting-back 0 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:16:57.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:16:57.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:16:57.479 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:16:57.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:16:57.480 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:16:57.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:16:57.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:16:57.481 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:16:57.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:57.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:16:57.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:16:57.483 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:57.483 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:16:57.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:16:57.484 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:16:57.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:16:57.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:57.486 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478073 2024-04-25T09:16:57.486 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.478073 2024-04-25T09:16:57.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:16:57.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:16:57.488 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:16:57.488 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:16:57.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:16:57.498 INFO:tasks.workunit:Running workunit osd/repro_long_log.sh... 2024-04-25T09:16:57.499 DEBUG:teuthology.orchestra.run.smithi088:workunit test osd/repro_long_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh 2024-04-25T09:16:57.550 INFO:tasks.workunit.client.0.smithi088.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:16:57.555 INFO:tasks.workunit.client.0.smithi088.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:16:57.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:16:57.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:16:57.555 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/repro-long-log 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:23: run: local dir=td/repro-long-log 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:24: run: shift 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: export CEPH_MON=127.0.0.1:7100 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: CEPH_MON=127.0.0.1:7100 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:27: run: export CEPH_ARGS 2024-04-25T09:16:57.556 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: uuidgen 2024-04-25T09:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: CEPH_ARGS+='--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none ' 2024-04-25T09:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-04-25T09:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: set 2024-04-25T09:16:57.557 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:16:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: local 'funcs=TEST_repro_long_log1 2024-04-25T09:16:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_repro_long_log2 2024-04-25T09:16:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_trim_max_entries 2024-04-25T09:16:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:TEST_trim_max_entries_with_dups' 2024-04-25T09:16:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:16:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:16:57.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:16:57.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:16:57.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:16:57.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:16:57.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:16:57.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:16:57.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:16:57.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:16:57.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:16:57.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:57.564 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:16:57.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:16:57.565 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:16:57.565 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:16:57.566 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:16:57.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:16:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:16:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:57.568 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:16:57.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:16:57.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:16:57.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:16:57.569 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:16:57.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:16:57.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:57.570 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:16:57.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.493765 2024-04-25T09:16:57.571 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:16:57.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:16:57.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:16:57.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:16:57.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:16:57.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log1 td/repro-long-log 2024-04-25T09:16:57.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:97: TEST_repro_long_log1: local dir=td/repro-long-log 2024-04-25T09:16:57.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:99: TEST_repro_long_log1: setup_log_test td/repro-long-log 2024-04-25T09:16:57.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:16:57.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:16:57.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:16:57.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:16:57.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:16:57.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:16:57.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:16:57.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:16:57.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:16:58.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:16:58.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:16:58.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:16:58.064 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:16:58.064 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:58.064 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:16:58.064 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:16:58.064 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:16:58.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:16:58.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:16:58.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:16:58.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:16:58.276 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:16:58.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:16:58.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:16:58.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:16:58.277 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:16:58.277 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:16:58.278 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:58.278 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:16:58.278 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493765/ceph-mon.a.asok 2024-04-25T09:16:58.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:16:58.278 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493765/ceph-mon.a.asok config get fsid 2024-04-25T09:16:58.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:16:58.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:16:58.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:16:58.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:16:58.396 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:16:58.396 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:16:58.396 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:16:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:16:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:16:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:16:58.397 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493765/ceph-mon.a.asok 2024-04-25T09:16:58.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:16:58.397 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493765/ceph-mon.a.asok config get mon_host 2024-04-25T09:16:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:16:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:16:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:16:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:16:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:16:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:16:58.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:16:59.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:16:59.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:16:59.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:16:59.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:16:59.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:59.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:16:59.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:16:59.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:16:59.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:16:59.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:16:59.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:16:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:16:59.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:16:59.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:16:59.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:16:59.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:16:59.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:16:59.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:16:59.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:16:59.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:16:59.419 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:16:59.420 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 de4985ec-4923-4439-b0d7-ab6f053cb064 2024-04-25T09:16:59.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=de4985ec-4923-4439-b0d7-ab6f053cb064 2024-04-25T09:16:59.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 de4985ec-4923-4439-b0d7-ab6f053cb064' 2024-04-25T09:16:59.420 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:16:59.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLHypmptDIGhAAdLjQ3dw/DJNQ+xY3P8UlCQ== 2024-04-25T09:16:59.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLHypmptDIGhAAdLjQ3dw/DJNQ+xY3P8UlCQ=="}' 2024-04-25T09:16:59.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new de4985ec-4923-4439-b0d7-ab6f053cb064 -i td/repro-long-log/0/new.json 2024-04-25T09:16:59.710 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:16:59.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:16:59.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLHypmptDIGhAAdLjQ3dw/DJNQ+xY3P8UlCQ== --osd-uuid de4985ec-4923-4439-b0d7-ab6f053cb064 2024-04-25T09:16:59.757 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:59.768+0000 7f03a96b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:59.779 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:59.792+0000 7f03a96b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:59.781 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:59.796+0000 7f03a96b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:59.781 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:59.796+0000 7f03a96b27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:16:59.781 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:59.796+0000 7f03a96b27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:16:59.781 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:59.796+0000 7f03a96b27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:16:59.820 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:16:59.832+0000 7f03a96b27c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:17:06.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:17:06.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:17:06.040 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:17:06.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:17:06.041 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:17:06.604 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:17:06.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:17:06.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:17:06.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:17:06.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:17:06.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:17:06.633 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:06.648+0000 7f4bdba3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:06.677 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:06.688+0000 7f4bdba3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:06.678 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:06.692+0000 7f4bdba3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:07.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:17:07.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:17:07.038 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:17:07.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:17:07.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:17:07.039 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:17:07.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:07.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:17:07.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:07.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:07.242 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:07.256+0000 7f4bdba3d7c0 -1 Falling back to public interface 2024-04-25T09:17:07.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:08.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:08.481 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:08.482 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:17:08.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:17:08.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:08.482 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:08.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:09.797 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:09.812+0000 7f4bdba3d7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:17:09.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:09.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:09.895 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:17:09.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:17:09.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:09.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:10.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:11.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:11.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:11.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:17:11.513 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:17:11.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:11.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:11.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:12.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:12.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:12.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:17:12.949 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:17:12.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:12.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:13.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:14.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:14.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:14.380 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:17:14.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:17:14.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:14.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:14.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:15.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:15.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:15.835 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:17:15.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:17:15.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:15.835 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:16.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:17.281 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:17.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:17.282 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:17:17.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:17:17.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:17.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:17.751 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:18.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:18.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:18.754 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:17:18.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:17:18.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:18.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:19.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:20.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:20.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:20.175 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:17:20.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:17:20.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:20.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:20.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:21.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:21.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:21.686 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:17:21.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:17:21.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:21.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:22.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:23.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:23.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:23.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:17:23.149 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:17:23.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:23.149 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:23.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:24.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:24.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:24.704 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:17:24.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:17:24.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:24.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:25.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:26.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:26.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:26.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:17:26.436 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:17:26.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:26.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:27.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:28.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:28.713 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:17:28.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:17:28.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:28.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:29.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:30.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:30.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:30.226 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:17:30.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:17:30.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:30.227 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:30.773 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:31.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:31.775 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:31.775 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:17:31.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:17:31.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:31.776 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:32.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:33.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:33.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:33.668 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:17:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:17:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:33.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:34.166 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:35.167 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:35.167 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:17:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:17:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:35.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:35.601 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1732496388,v1:127.0.0.1:6803/1732496388] [v2:127.0.0.1:6804/1732496388,v1:127.0.0.1:6805/1732496388] exists,up de4985ec-4923-4439-b0d7-ab6f053cb064 2024-04-25T09:17:35.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:17:35.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:17:35.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:17:35.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:17:35.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:17:35.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:17:35.603 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:17:35.604 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:17:35.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:17:35.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:17:35.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:17:35.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:17:35.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:17:35.606 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:17:35.606 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=afecc612-9e88-410a-94c5-a02e197e4820 2024-04-25T09:17:35.607 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 afecc612-9e88-410a-94c5-a02e197e4820 2024-04-25T09:17:35.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 afecc612-9e88-410a-94c5-a02e197e4820' 2024-04-25T09:17:35.607 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:17:35.620 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvHypmJy30JRAAEb4+k41XaVF2W05ad+Jndw== 2024-04-25T09:17:35.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvHypmJy30JRAAEb4+k41XaVF2W05ad+Jndw=="}' 2024-04-25T09:17:35.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new afecc612-9e88-410a-94c5-a02e197e4820 -i td/repro-long-log/1/new.json 2024-04-25T09:17:36.191 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:17:36.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:17:36.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvHypmJy30JRAAEb4+k41XaVF2W05ad+Jndw== --osd-uuid afecc612-9e88-410a-94c5-a02e197e4820 2024-04-25T09:17:36.243 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:36.256+0000 7f97e7a257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:36.246 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:36.260+0000 7f97e7a257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:36.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:36.260+0000 7f97e7a257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:36.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:36.260+0000 7f97e7a257c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:17:36.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:36.260+0000 7f97e7a257c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:17:36.248 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:36.260+0000 7f97e7a257c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:17:36.287 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:36.300+0000 7f97e7a257c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:17:42.984 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:17:42.985 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:17:42.986 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T09:17:42.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:17:42.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:17:43.544 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:17:43.546 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:17:43.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:17:43.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:17:43.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:17:43.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:17:43.573 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:43.588+0000 7fc9c9bc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:43.575 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:43.588+0000 7fc9c9bc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:43.576 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:43.588+0000 7fc9c9bc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:43.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:17:43.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:17:43.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:17:43.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:17:43.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:17:43.959 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:43.959 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:17:43.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:17:43.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:43.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:44.122 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:44.136+0000 7fc9c9bc67c0 -1 Falling back to public interface 2024-04-25T09:17:44.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:45.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:45.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:45.365 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:17:45.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:17:45.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:45.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:45.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:46.735 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:17:46.747+0000 7fc9c9bc67c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:17:46.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:46.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:46.757 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:17:46.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:17:46.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:46.758 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:47.273 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:48.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:48.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:48.276 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:17:48.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:17:48.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:48.276 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:48.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:49.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:49.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:49.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:17:49.699 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:17:49.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:49.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:50.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:51.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:17:51.132 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:17:51.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:51.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:51.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:52.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:52.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:52.723 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:17:52.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:17:52.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:52.724 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:53.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:54.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:54.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:54.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:17:54.210 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:17:54.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:54.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:54.757 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:55.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:55.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:55.759 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:17:55.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:17:55.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:55.760 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:56.182 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:57.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:57.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:57.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:17:57.184 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:17:57.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:57.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:57.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:58.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:58.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:58.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:17:58.647 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:17:58.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:58.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:17:59.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:00.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:00.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:00.101 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:18:00.101 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:18:00.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:00.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:00.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:01.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:01.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:01.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:18:01.611 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:18:01.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:01.611 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:02.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:03.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:03.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:03.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:18:03.028 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:18:03.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:03.029 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:03.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:04.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:04.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:04.791 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:18:04.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:18:04.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:04.792 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:05.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:06.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:06.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:06.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:18:06.863 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:18:06.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:06.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:07.784 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:08.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:08.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:08.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:18:08.787 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:18:08.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:08.787 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:09.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:10.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:10.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:10.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:18:10.470 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:18:10.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:10.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:11.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:12.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:12.295 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:12.295 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:18:12.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:18:12.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:12.296 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:12.707 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1944119484,v1:127.0.0.1:6811/1944119484] [v2:127.0.0.1:6812/1944119484,v1:127.0.0.1:6813/1944119484] exists,up afecc612-9e88-410a-94c5-a02e197e4820 2024-04-25T09:18:12.710 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:18:12.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:18:12.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:18:12.713 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:18:12.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:18:12.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=24f0df67-4a20-4867-b674-67d244ee3c53 2024-04-25T09:18:12.714 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 24f0df67-4a20-4867-b674-67d244ee3c53 2024-04-25T09:18:12.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 24f0df67-4a20-4867-b674-67d244ee3c53' 2024-04-25T09:18:12.714 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:18:12.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUHypmmvZmLBAAjiiUnJzvaZGJhxYQEBhOew== 2024-04-25T09:18:12.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUHypmmvZmLBAAjiiUnJzvaZGJhxYQEBhOew=="}' 2024-04-25T09:18:12.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 24f0df67-4a20-4867-b674-67d244ee3c53 -i td/repro-long-log/2/new.json 2024-04-25T09:18:13.253 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:18:13.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:18:13.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUHypmmvZmLBAAjiiUnJzvaZGJhxYQEBhOew== --osd-uuid 24f0df67-4a20-4867-b674-67d244ee3c53 2024-04-25T09:18:13.302 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:13.315+0000 7fac4bd037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:13.332 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:13.343+0000 7fac4bd037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:13.333 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:13.347+0000 7fac4bd037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:13.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:13.347+0000 7fac4bd037c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:18:13.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:13.347+0000 7fac4bd037c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:18:13.334 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:13.347+0000 7fac4bd037c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:18:13.467 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:13.483+0000 7fac4bd037c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:18:19.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:18:19.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:18:19.928 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T09:18:19.928 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:18:19.929 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:18:20.474 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:18:20.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:18:20.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:18:20.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:18:20.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:18:20.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:18:20.504 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:20.515+0000 7fbe27f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:20.541 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:20.551+0000 7fbe27f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:20.542 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:20.555+0000 7fbe27f6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:20.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:18:20.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:18:20.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:18:20.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:18:20.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:18:20.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:20.898 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:18:20.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:18:20.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:20.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:21.107 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:21.119+0000 7fbe27f6d7c0 -1 Falling back to public interface 2024-04-25T09:18:21.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:22.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:22.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:22.316 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:18:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:18:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:22.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:22.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:23.612 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:18:23.623+0000 7fbe27f6d7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:18:23.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:23.737 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:18:23.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:23.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:18:23.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:23.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:24.297 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:25.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:25.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:25.299 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:18:25.299 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:18:25.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:25.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:25.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:26.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:26.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:26.815 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:18:26.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:18:26.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:26.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:27.344 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:28.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:28.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:28.346 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:18:28.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:18:28.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:28.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:28.923 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:29.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:29.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:29.924 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:18:29.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:18:29.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:29.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:30.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:31.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:31.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:31.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:18:31.524 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:18:31.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:31.525 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:31.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:32.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:32.947 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:32.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:18:32.948 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:18:32.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:32.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:33.499 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:34.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:34.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:34.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:18:34.502 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:18:34.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:34.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:35.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:36.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:36.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:36.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:18:36.111 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:18:36.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:36.112 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:36.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:37.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:37.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:37.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:18:37.650 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:18:37.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:37.651 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:38.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:39.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:39.314 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:39.314 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:18:39.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:18:39.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:39.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:40.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:41.540 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:41.541 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:18:41.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:41.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:18:41.541 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:41.542 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:42.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:43.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:43.341 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:43.341 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:18:43.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:18:43.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:43.342 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:44.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:45.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:45.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:45.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:18:45.115 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:18:45.115 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:45.116 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:45.931 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:46.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:46.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:46.933 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:18:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:18:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:46.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:47.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:48.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:48.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:48.715 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:18:48.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:18:48.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:48.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:18:49.135 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4112376528,v1:127.0.0.1:6819/4112376528] [v2:127.0.0.1:6820/4112376528,v1:127.0.0.1:6821/4112376528] exists,up 24f0df67-4a20-4867-b674-67d244ee3c53 2024-04-25T09:18:49.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:18:49.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:18:49.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:18:49.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:18:49.842 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T09:18:49.861 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:18:49.862 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:18:50.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:18:50.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:18:50.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:18:50.580 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:18:50.607 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:18:50.636 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:18:50.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:18:50.785 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:18:50.812 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:18:50.839 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:18:50.853 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:18:50.988 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:18:51.019 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:18:51.051 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:18:51.065 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:18:51.206 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:18:51.235 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:18:51.266 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:18:51.280 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:18:51.507 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:18:51.539 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:18:51.577 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:18:51.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:18:51.592 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:18:51.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:51.593 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:51.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:51.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:52.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:52.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:52.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:52.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:52.625 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:52.626 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:52.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:52.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:52.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:52.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:53.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:53.094 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:53.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:53.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:53.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:53.470 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:53.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:53.596 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:53.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:53.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:53.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:53.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:54.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:54.037 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:54.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:54.157 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:54.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:54.275 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:54.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:54.391 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:54.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:54.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:54.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:54.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:18:54.820 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:18:54.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:18:54.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:18:54.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:18:54.821 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:18:54.963 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836507 2024-04-25T09:18:54.987 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672974 2024-04-25T09:18:55.011 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509446 2024-04-25T09:18:55.026 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:18:58.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:18:58.028 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:18:58.191 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:18:58.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:18:58.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:18:58.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:18:58.346 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:18:58.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:18:58.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:18:58.506 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:18:58.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:18:58.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:18:58.507 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:18:58.659 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:18:58.660 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:18:58.832 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:18:58.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:58.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:18:58.945 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:18:58.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:58.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:18:59.062 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:18:59.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:59.066 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:18:59.179 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:18:59.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:59.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:18:59.297 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:18:59.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:59.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:18:59.413 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:18:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:59.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:18:59.539 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:18:59.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:59.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:18:59.723 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:18:59.726 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:59.727 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:18:59.874 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:18:59.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:18:59.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:00.033 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:00.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:00.036 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:00.158 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:00.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:00.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:00.284 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:00.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:00.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:00.417 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:00.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:00.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:00.593 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:00.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:00.597 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:00.728 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:00.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:00.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:00.845 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:00.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:00.849 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:00.962 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:00.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:00.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:01.079 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:01.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:01.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:01.205 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:01.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:01.209 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:01.322 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:01.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:19:01.325 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:19:01.489 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:19:01.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:19:01.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:19:01.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:19:01.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:19:01.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:19:01.635 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836511 2024-04-25T09:19:01.659 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672977 2024-04-25T09:19:01.683 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509449 2024-04-25T09:19:01.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:19:04.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:19:04.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:19:04.864 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T09:19:04.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:19:04.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:19:04.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:19:05.022 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T09:19:05.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:19:05.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:19:05.171 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:19:05.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:19:05.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:19:05.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:19:05.328 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:19:05.328 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:101: TEST_repro_long_log1: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:19:05.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:102: TEST_repro_long_log1: test_log_size 1.0 22 2024-04-25T09:19:05.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:19:05.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=22 2024-04-25T09:19:05.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:19:05.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:19:05.583 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836514 2024-04-25T09:19:05.606 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672980 2024-04-25T09:19:05.631 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509452 2024-04-25T09:19:05.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:19:08.648 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:19:08.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:19:08.814 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T09:19:08.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:19:08.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:19:08.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 22 2024-04-25T09:19:08.968 INFO:tasks.workunit.client.0.smithi088.stdout:22 2024-04-25T09:19:08.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:19:08.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:19:09.123 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:19:09.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:19:09.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:19:09.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:19:09.273 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:19:09.274 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:19:09.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:19:09.382 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:19:09.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:19:09.383 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:19:09.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:19:09.384 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:19:09.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:19:09.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:09.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:19:09.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:19:09.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:09.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:19:09.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:19:09.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:19:09.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:19:09.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:09.403 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:09.403 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:19:09.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:19:09.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:19:09.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:19:09.406 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:19:09.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:19:09.407 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:19:09.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:19:09.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:19:09.408 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:19:09.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:09.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:19:09.409 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:19:09.409 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:09.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:19:09.415 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:19:09.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:09.416 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:09.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.493765 2024-04-25T09:19:09.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:19:09.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log2 td/repro-long-log 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:107: TEST_repro_long_log2: local dir=td/repro-long-log 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:109: TEST_repro_long_log2: setup_log_test td/repro-long-log 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:19:09.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:19:09.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:19:09.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:09.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:09.879 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:09.879 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:09.879 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:09.879 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:19:09.880 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:19:10.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:19:10.092 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:19:10.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:19:10.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:19:10.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:19:10.093 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:19:10.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:19:10.093 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:19:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:19:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:19:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:10.094 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493765/ceph-mon.a.asok 2024-04-25T09:19:10.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:19:10.095 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493765/ceph-mon.a.asok config get fsid 2024-04-25T09:19:10.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:19:10.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:19:10.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:19:10.212 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:19:10.213 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:19:10.213 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:19:10.213 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:19:10.213 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:19:10.213 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:19:10.214 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:10.214 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:10.214 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493765/ceph-mon.a.asok 2024-04-25T09:19:10.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:19:10.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493765/ceph-mon.a.asok config get mon_host 2024-04-25T09:19:10.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:19:10.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:19:10.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:19:10.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:19:10.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:19:10.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:19:10.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:19:11.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:19:11.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:11.247 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:11.248 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:11.248 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:11.248 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:11.248 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:19:11.249 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:19:11.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:19:11.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:19:11.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:19:11.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:19:11.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:19:11.288 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:19:11.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:19:11.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:19:11.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:19:11.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:19:11.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:19:11.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:19:11.290 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:19:11.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a01c60a1-db5f-4e9f-8269-c2008582324d 2024-04-25T09:19:11.291 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 a01c60a1-db5f-4e9f-8269-c2008582324d 2024-04-25T09:19:11.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a01c60a1-db5f-4e9f-8269-c2008582324d' 2024-04-25T09:19:11.292 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:19:11.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPICpm738oExAAMNj2c4sqD70rJ4iBd/ccIw== 2024-04-25T09:19:11.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPICpm738oExAAMNj2c4sqD70rJ4iBd/ccIw=="}' 2024-04-25T09:19:11.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a01c60a1-db5f-4e9f-8269-c2008582324d -i td/repro-long-log/0/new.json 2024-04-25T09:19:11.635 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:19:11.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:19:11.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPICpm738oExAAMNj2c4sqD70rJ4iBd/ccIw== --osd-uuid a01c60a1-db5f-4e9f-8269-c2008582324d 2024-04-25T09:19:11.688 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:11.698+0000 7fdab2c6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:11.712 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:11.722+0000 7fdab2c6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:11.713 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:11.726+0000 7fdab2c6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:11.714 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:11.726+0000 7fdab2c6f7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:19:11.714 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:11.726+0000 7fdab2c6f7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:19:11.714 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:11.726+0000 7fdab2c6f7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:19:11.752 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:11.762+0000 7fdab2c6f7c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:19:18.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:19:18.084 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:19:18.085 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:19:18.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:19:18.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:19:18.620 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:19:18.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:19:18.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:19:18.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:19:18.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:19:18.621 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:19:18.649 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:18.662+0000 7f390c6d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:18.651 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:18.662+0000 7f390c6d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:18.653 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:18.666+0000 7f390c6d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:19:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:19:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:19:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:19:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:19:19.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:19:19.019 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:19:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:19.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:19.187 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:19.202+0000 7f390c6d37c0 -1 Falling back to public interface 2024-04-25T09:19:19.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:20.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:20.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:20.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:19:20.458 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:19:20.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:20.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:20.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:21.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:21.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:21.863 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:19:21.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:19:21.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:21.864 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:21.908 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:21.926+0000 7f390c6d37c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:19:22.498 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:23.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:23.500 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:23.501 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:19:23.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:19:23.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:23.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:23.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:24.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:24.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:24.937 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:19:24.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:19:24.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:24.938 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:25.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:26.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:26.414 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:26.414 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:19:26.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:19:26.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:26.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:26.935 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:27.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:27.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:27.938 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:19:27.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:19:27.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:27.939 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:28.365 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:29.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:29.366 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:29.367 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:19:29.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:19:29.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:29.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:29.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:30.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:30.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:30.828 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:19:30.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:19:30.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:30.829 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:31.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:32.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:32.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:32.262 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:19:32.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:19:32.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:32.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:32.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:33.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:33.847 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:33.847 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:19:33.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:19:33.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:33.848 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:34.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:35.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:35.270 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:35.271 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:19:35.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:19:35.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:35.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:35.940 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:36.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:36.942 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:36.942 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:19:36.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:19:36.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:36.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:37.728 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:38.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:38.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:38.731 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:19:38.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:19:38.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:38.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:39.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:40.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:40.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:40.891 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:19:40.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:19:40.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:40.892 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:41.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:42.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:42.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:42.469 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:19:42.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:19:42.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:42.471 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:42.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:43.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:43.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:43.973 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:19:43.973 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:19:43.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:43.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:44.444 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:45.445 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:45.446 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:45.446 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:19:45.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:19:45.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:45.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:46.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:47.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:47.047 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:19:47.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:19:47.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:47.047 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:47.454 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2528187960,v1:127.0.0.1:6803/2528187960] [v2:127.0.0.1:6804/2528187960,v1:127.0.0.1:6805/2528187960] exists,up a01c60a1-db5f-4e9f-8269-c2008582324d 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:19:47.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:19:47.459 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:19:47.460 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:19:47.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5272798-a043-4d03-b3e0-95894d5f4b24 2024-04-25T09:19:47.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c5272798-a043-4d03-b3e0-95894d5f4b24' 2024-04-25T09:19:47.461 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 c5272798-a043-4d03-b3e0-95894d5f4b24 2024-04-25T09:19:47.462 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:19:47.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzICpmM455HRAApje70irJd9vXHXBXQQ/cyA== 2024-04-25T09:19:47.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzICpmM455HRAApje70irJd9vXHXBXQQ/cyA=="}' 2024-04-25T09:19:47.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5272798-a043-4d03-b3e0-95894d5f4b24 -i td/repro-long-log/1/new.json 2024-04-25T09:19:48.001 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:19:48.017 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:19:48.018 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzICpmM455HRAApje70irJd9vXHXBXQQ/cyA== --osd-uuid c5272798-a043-4d03-b3e0-95894d5f4b24 2024-04-25T09:19:48.051 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:48.065+0000 7fc7645537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:48.114 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:48.129+0000 7fc7645537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:48.116 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:48.133+0000 7fc7645537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:48.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:48.133+0000 7fc7645537c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:19:48.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:48.133+0000 7fc7645537c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:19:48.117 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:48.133+0000 7fc7645537c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:19:48.238 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:48.253+0000 7fc7645537c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:19:54.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:19:54.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:19:54.719 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T09:19:54.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:19:54.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:19:55.271 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:19:55.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:19:55.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:19:55.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:19:55.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:19:55.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:19:55.301 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:55.317+0000 7f4691f657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:55.303 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:55.317+0000 7f4691f657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:55.305 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:55.321+0000 7f4691f657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:55.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:19:55.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:19:55.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:19:55.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:19:55.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:19:55.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:55.668 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:19:55.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:19:55.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:55.668 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:55.847 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:55.861+0000 7f4691f657c0 -1 Falling back to public interface 2024-04-25T09:19:56.108 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:57.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:57.110 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:19:57.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:57.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:19:57.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:57.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:57.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:58.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:58.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:58.524 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:19:58.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:19:58.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:58.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:58.738 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:19:58.753+0000 7f4691f657c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:19:58.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:59.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:59.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:59.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:19:59.982 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:19:59.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:59.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:00.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:01.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:01.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:01.418 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:20:01.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:20:01.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:01.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:01.958 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:02.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:02.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:02.960 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:20:02.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:20:02.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:02.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:03.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:04.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:04.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:04.382 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:20:04.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:20:04.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:04.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:04.994 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:05.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:05.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:05.996 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:20:05.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:20:05.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:05.997 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:06.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:07.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:07.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:07.415 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:20:07.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:20:07.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:07.416 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:07.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:08.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:08.965 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:08.966 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:20:08.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:20:08.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:08.966 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:09.392 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:10.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:10.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:10.394 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:20:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:20:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:10.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:10.967 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:11.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:11.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:11.969 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:20:11.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:20:11.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:11.970 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:12.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:13.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:13.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:13.502 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:20:13.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:20:13.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:13.503 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:14.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:15.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:15.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:15.248 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:20:15.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:20:15.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:15.248 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:16.461 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:17.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:17.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:17.463 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:20:17.464 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:20:17.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:17.464 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:18.283 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:19.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:19.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:19.285 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:20:19.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:20:19.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:19.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:20.189 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:21.191 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:20:21.191 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:20:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:21.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:22.393 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:23.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:23.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:23.395 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:20:23.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:20:23.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:23.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:24.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:25.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:25.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:25.046 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:20:25.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:20:25.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:25.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:25.461 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3665830980,v1:127.0.0.1:6811/3665830980] [v2:127.0.0.1:6812/3665830980,v1:127.0.0.1:6813/3665830980] exists,up c5272798-a043-4d03-b3e0-95894d5f4b24 2024-04-25T09:20:25.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:20:25.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:20:25.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:20:25.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:20:25.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:20:25.465 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:20:25.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:20:25.467 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe6f5ae9-0999-40c2-b2d2-a72006e1446c 2024-04-25T09:20:25.468 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 fe6f5ae9-0999-40c2-b2d2-a72006e1446c 2024-04-25T09:20:25.469 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fe6f5ae9-0999-40c2-b2d2-a72006e1446c' 2024-04-25T09:20:25.469 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:20:25.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZICpmgc7lHRAABpfZERuEiCcMIMU3bgvu9A== 2024-04-25T09:20:25.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZICpmgc7lHRAABpfZERuEiCcMIMU3bgvu9A=="}' 2024-04-25T09:20:25.485 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe6f5ae9-0999-40c2-b2d2-a72006e1446c -i td/repro-long-log/2/new.json 2024-04-25T09:20:25.973 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:20:25.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:20:25.990 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZICpmgc7lHRAABpfZERuEiCcMIMU3bgvu9A== --osd-uuid fe6f5ae9-0999-40c2-b2d2-a72006e1446c 2024-04-25T09:20:26.024 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:26.036+0000 7f9c6833a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:26.082 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:26.096+0000 7f9c6833a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:26.083 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:26.096+0000 7f9c6833a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:26.096+0000 7f9c6833a7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:20:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:26.096+0000 7f9c6833a7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:20:26.084 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:26.096+0000 7f9c6833a7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:20:26.231 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:26.244+0000 7f9c6833a7c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:20:32.861 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:20:32.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:20:32.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:20:32.862 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T09:20:32.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:20:33.451 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:20:33.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:20:33.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:20:33.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:20:33.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:20:33.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:20:33.481 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:33.496+0000 7f63489a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:33.483 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:33.496+0000 7f63489a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:33.484 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:33.500+0000 7f63489a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:33.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:20:33.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:20:33.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:20:33.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:20:33.908 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:20:33.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:33.909 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:20:33.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:20:33.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:33.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:34.070 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:34.084+0000 7f63489a47c0 -1 Falling back to public interface 2024-04-25T09:20:34.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:35.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:35.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:35.318 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:20:35.319 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:20:35.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:35.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:35.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:36.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:36.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:36.743 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:20:36.744 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:20:36.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:36.744 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:36.795 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:20:36.808+0000 7f63489a47c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:20:37.317 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:38.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:38.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:38.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:20:38.319 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:20:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:38.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:38.752 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:39.753 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:39.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:39.754 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:20:39.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:20:39.754 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:39.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:40.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:41.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:41.264 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:41.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:20:41.265 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:20:41.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:41.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:41.716 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:42.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:42.718 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:42.718 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:20:42.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:20:42.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:42.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:43.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:44.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:44.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:44.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:20:44.123 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:20:44.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:44.124 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:44.615 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:45.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:45.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:45.617 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:20:45.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:20:45.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:45.617 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:46.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:47.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:47.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:47.204 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:20:47.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:20:47.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:47.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:47.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:48.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:48.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:48.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:20:48.678 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:20:48.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:48.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:49.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:50.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:20:50.145 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:20:50.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:50.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:50.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:51.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:51.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:51.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:20:51.688 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:20:51.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:51.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:52.282 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:53.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:53.285 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:53.285 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:20:53.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:20:53.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:53.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:54.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:55.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:55.113 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:55.114 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:20:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:20:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:55.114 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:55.896 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:56.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:56.898 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:56.898 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:20:56.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:20:56.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:56.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:57.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:58.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:58.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:58.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:20:58.732 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:20:58.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:58.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:59.934 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:00.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:00.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:00.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:21:00.936 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:21:00.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:00.937 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:01.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:02.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:02.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:02.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:21:02.450 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:21:02.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:02.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:02.876 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/362360177,v1:127.0.0.1:6819/362360177] [v2:127.0.0.1:6820/362360177,v1:127.0.0.1:6821/362360177] exists,up fe6f5ae9-0999-40c2-b2d2-a72006e1446c 2024-04-25T09:21:02.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:21:02.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:21:02.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:21:02.877 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:21:03.412 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T09:21:03.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:21:03.431 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:21:03.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:21:03.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:21:03.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:21:04.164 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:21:04.205 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:21:04.246 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:21:04.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:21:04.451 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:21:04.480 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:21:04.509 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:21:04.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:21:04.781 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:21:04.819 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:21:04.857 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:21:04.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:21:05.018 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:21:05.045 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:21:05.072 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:21:05.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:21:05.225 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:21:05.255 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:21:05.283 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:21:05.298 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:21:05.299 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:21:05.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:05.300 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:05.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:05.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:05.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:05.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:06.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:06.106 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:06.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:06.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:06.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:06.408 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:06.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:06.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:06.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:06.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:06.785 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:06.786 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:06.911 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:06.912 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:07.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:07.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:07.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:07.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:07.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:07.288 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:07.405 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:07.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:07.582 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:07.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:07.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:07.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:07.824 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:07.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:07.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:08.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:08.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:08.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:08.192 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:21:08.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:21:08.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:21:08.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:21:08.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:21:08.302 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:21:08.450 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836503 2024-04-25T09:21:08.474 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672979 2024-04-25T09:21:08.498 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509448 2024-04-25T09:21:08.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:21:11.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:11.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:21:11.679 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:21:11.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:11.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:21:11.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:21:11.837 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:21:11.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:11.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:21:11.992 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:21:11.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:11.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:21:11.993 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:21:12.142 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:21:12.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:21:12.277 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:21:12.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:12.279 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:12.416 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:12.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:12.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:12.568 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:12.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:12.572 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:12.718 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:12.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:12.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:12.835 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:12.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:12.838 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:12.952 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:12.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:12.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:13.094 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:13.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:13.098 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:13.203 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:13.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:13.206 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:13.345 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:13.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:13.454 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:13.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:13.458 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:13.605 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:13.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:13.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:13.714 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:13.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:13.717 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:13.822 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:13.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:13.826 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:13.939 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:13.943 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:13.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:14.115 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:14.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:14.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:14.224 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:14.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:14.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:14.350 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:14.354 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:14.526 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:14.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:14.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:14.643 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:14.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:14.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:14.761 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:14.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:21:14.764 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:21:14.911 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:21:14.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:21:14.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:21:14.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:21:14.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:21:14.916 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:21:15.058 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836506 2024-04-25T09:21:15.082 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672982 2024-04-25T09:21:15.106 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509451 2024-04-25T09:21:15.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:21:18.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:18.123 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:21:18.290 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T09:21:18.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:18.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:21:18.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:21:18.447 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T09:21:18.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:18.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:21:18.607 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:21:18.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:18.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:21:18.608 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:21:18.760 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:21:18.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: ceph pg 1.0 query 2024-04-25T09:21:18.761 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: jq .info.stats.up_primary 2024-04-25T09:21:18.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: local PRIMARY=1 2024-04-25T09:21:18.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:111: TEST_repro_long_log2: kill_daemons td/repro-long-log TERM osd.1 2024-04-25T09:21:18.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:21:18.915 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:21:18.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:21:18.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:21:18.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:21:19.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:21:19.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: CEPH_ARGS='--osd-max-pg-log-entries=2 --osd-pg-log-dups-tracked=3 --no-mon-config' 2024-04-25T09:21:19.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: ceph-objectstore-tool --data-path td/repro-long-log/1 --pgid 1.0 --op trim-pg-log 2024-04-25T09:21:20.841 INFO:tasks.workunit.client.0.smithi088.stderr:Log bounds are: (19'20,19'41] 2024-04-25T09:21:20.842 INFO:tasks.workunit.client.0.smithi088.stdout:Removing keys 0000000019.00000000000000000021 - 0000000019.00000000000000000039 2024-04-25T09:21:20.871 INFO:tasks.workunit.client.0.smithi088.stderr:Finished trimming, now compacting... 2024-04-25T09:21:21.867 INFO:tasks.workunit.client.0.smithi088.stdout:Finished trimming pg log 2024-04-25T09:21:22.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:113: TEST_repro_long_log2: activate_osd td/repro-long-log 1 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repro-long-log 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:21:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:21:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:21:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:21:22.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:21:22.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:21:22.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:21:22.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:21:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:21:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:22.385 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:21:22.386 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:21:22.387 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:21:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T09:21:22.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:21:22.388 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repro-long-log/1/whoami 2024-04-25T09:21:22.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T09:21:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:21:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:21:22.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:21:22.419 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:22.431+0000 7ff8ad2c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:22.422 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:22.435+0000 7ff8ad2c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:22.423 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:22.435+0000 7ff8ad2c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T09:21:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:21:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:21:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:21:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:21:22.822 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:22.823 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:21:22.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:21:22.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:22.823 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:22.981 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:22.995+0000 7ff8ad2c47c0 -1 Falling back to public interface 2024-04-25T09:21:23.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:24.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:24.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:24.267 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:21:24.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:21:24.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:24.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:24.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:25.484 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:25.495+0000 7ff8ad2c47c0 -1 osd.1 19 log_to_monitors true 2024-04-25T09:21:25.699 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:25.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:25.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:21:25.700 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:21:25.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:25.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:26.151 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 23 up_thru 17 down_at 20 last_clean_interval [10,19) [v2:127.0.0.1:6810/4067408144,v1:127.0.0.1:6811/4067408144] [v2:127.0.0.1:6812/4067408144,v1:127.0.0.1:6813/4067408144] exists,up c5272798-a043-4d03-b3e0-95894d5f4b24 2024-04-25T09:21:26.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:21:26.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:21:26.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:21:26.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:114: TEST_repro_long_log2: wait_for_clean 2024-04-25T09:21:26.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:21:26.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:21:26.152 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:21:26.152 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:21:26.153 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:21:26.153 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:21:26.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:21:26.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:21:26.153 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:21:26.245 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:21:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:21:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:21:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:21:26.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:21:26.246 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:21:26.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:21:26.671 INFO:tasks.workunit.client.0.smithi088.stderr:1 2024-04-25T09:21:26.671 INFO:tasks.workunit.client.0.smithi088.stderr:2' 2024-04-25T09:21:26.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:21:26.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:21:26.672 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:21:26.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T09:21:26.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T09:21:26.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T09:21:26.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:21:26.839 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:21:27.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=98784247811 2024-04-25T09:21:27.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 98784247811 2024-04-25T09:21:27.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-98784247811' 2024-04-25T09:21:27.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:21:27.063 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:21:27.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-04-25T09:21:27.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-04-25T09:21:27.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-98784247811 2-64424509455' 2024-04-25T09:21:27.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:21:27.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T09:21:27.218 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:21:27.220 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:21:27.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T09:21:27.220 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:21:27.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T09:21:27.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T09:21:27.221 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.0 seq 21474836511 2024-04-25T09:21:27.222 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:21:27.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836511 2024-04-25T09:21:27.643 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:21:28.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:21:28.645 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:21:29.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-04-25T09:21:29.068 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:21:29.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:21:29.069 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-98784247811 2024-04-25T09:21:29.070 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:21:29.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-98784247811 2024-04-25T09:21:29.071 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:21:29.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=98784247811 2024-04-25T09:21:29.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 98784247811' 2024-04-25T09:21:29.072 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.1 seq 98784247811 2024-04-25T09:21:29.072 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:21:29.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 98784247811 -lt 98784247811 2024-04-25T09:21:29.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:21:29.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-04-25T09:21:29.513 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:21:29.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:21:29.515 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-04-25T09:21:29.516 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:21:29.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-04-25T09:21:29.517 INFO:tasks.workunit.client.0.smithi088.stdout:waiting osd.2 seq 64424509455 2024-04-25T09:21:29.517 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-25T09:21:29.517 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:21:29.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509456 -lt 64424509455 2024-04-25T09:21:29.956 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:21:29.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:21:29.957 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:21:30.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:21:30.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:21:30.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:21:30.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:21:30.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:21:30.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:21:30.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:21:30.494 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:21:30.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:21:30.917 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:21:30.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:21:30.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:21:31.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:21:31.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:21:31.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:21:31.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:21:31.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:21:31.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:21:31.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:21:31.527 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:21:31.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:21:31.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:21:31.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:21:31.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:21:31.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:21:31.529 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:21:31.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:21:31.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:21:31.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:21:31.918 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:21:32.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:21:32.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:21:32.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:21:32.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:115: TEST_repro_long_log2: test_log_size 1.0 21 18 2024-04-25T09:21:32.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:21:32.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:21:32.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-04-25T09:21:32.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:21:32.614 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836514 2024-04-25T09:21:32.636 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 98784247814 2024-04-25T09:21:32.659 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509459 2024-04-25T09:21:32.672 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:21:35.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:35.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:21:35.841 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:21:35.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:35.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:21:35.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:21:36.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:36.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:21:36.167 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:21:36.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:21:36.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:21:36.168 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-04-25T09:21:36.322 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:21:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:21:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:21:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:21:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:21:36.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:21:36.323 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:21:36.323 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:21:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:21:36.324 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:21:36.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:21:36.633 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:21:36.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:21:36.635 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:21:36.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:21:36.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:21:36.636 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:21:36.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:21:36.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:21:36.638 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:21:36.638 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:21:36.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:21:36.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:21:36.641 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:21:36.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:21:36.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:36.664 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:36.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:21:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:21:36.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:21:36.667 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:21:36.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:21:36.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:21:36.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:21:36.669 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:21:36.670 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:21:36.671 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:21:36.671 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:21:36.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:21:36.673 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:21:36.673 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:21:36.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:21:36.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:21:36.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:21:36.674 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:21:36.675 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:21:36.676 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:21:36.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:21:36.678 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:21:36.678 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:36.678 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:36.678 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:21:36.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:21:36.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:21:36.679 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:21:36.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:21:36.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:36.681 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:36.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.493765 2024-04-25T09:21:36.684 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries td/repro-long-log 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:120: TEST_trim_max_entries: local dir=td/repro-long-log 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:122: TEST_trim_max_entries: setup_log_test td/repro-long-log 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:21:36.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:21:36.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:21:37.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:21:37.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:21:37.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:21:37.172 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:21:37.172 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:37.172 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:37.173 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:21:37.173 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:21:37.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:21:37.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:21:37.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:21:37.384 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:21:37.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:21:37.385 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:21:37.385 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:21:37.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:21:37.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:21:37.386 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:21:37.386 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:37.386 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:37.386 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493765/ceph-mon.a.asok 2024-04-25T09:21:37.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:21:37.387 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493765/ceph-mon.a.asok config get fsid 2024-04-25T09:21:37.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:21:37.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:21:37.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:21:37.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:21:37.503 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:21:37.503 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:21:37.503 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:21:37.504 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:21:37.504 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:21:37.504 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:37.505 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:37.505 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493765/ceph-mon.a.asok 2024-04-25T09:21:37.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:21:37.505 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493765/ceph-mon.a.asok config get mon_host 2024-04-25T09:21:37.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:21:37.613 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:21:37.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:21:37.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:21:37.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:21:37.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:21:37.614 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:21:38.527 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:21:38.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:21:38.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:21:38.528 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:21:38.528 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:38.528 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:38.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:21:38.528 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:21:38.530 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:21:38.559 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:21:38.560 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:21:38.561 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:21:38.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:21:38.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:21:38.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:21:38.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:21:38.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:21:38.562 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:21:38.562 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=41754d99-56eb-45bf-8190-0ad3ef6c2fe7 2024-04-25T09:21:38.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 41754d99-56eb-45bf-8190-0ad3ef6c2fe7' 2024-04-25T09:21:38.563 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 41754d99-56eb-45bf-8190-0ad3ef6c2fe7 2024-04-25T09:21:38.563 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:21:38.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiICpmTJhyIxAAt7Jqv0EBQAho6sQvxZyvJg== 2024-04-25T09:21:38.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiICpmTJhyIxAAt7Jqv0EBQAho6sQvxZyvJg=="}' 2024-04-25T09:21:38.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 41754d99-56eb-45bf-8190-0ad3ef6c2fe7 -i td/repro-long-log/0/new.json 2024-04-25T09:21:38.878 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:21:38.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:21:38.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiICpmTJhyIxAAt7Jqv0EBQAho6sQvxZyvJg== --osd-uuid 41754d99-56eb-45bf-8190-0ad3ef6c2fe7 2024-04-25T09:21:38.929 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:38.943+0000 7f9f1b6f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:38.946 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:38.959+0000 7f9f1b6f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:38.947 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:38.959+0000 7f9f1b6f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:38.948 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:38.963+0000 7f9f1b6f27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:21:38.948 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:38.963+0000 7f9f1b6f27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:21:38.948 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:38.963+0000 7f9f1b6f27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:21:38.986 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:38.999+0000 7f9f1b6f27c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:21:44.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:21:44.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:21:44.888 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:21:44.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:21:44.888 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:21:45.438 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:21:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:21:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:21:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:21:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:21:45.439 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:21:45.462 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:45.475+0000 7f3e56a067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:45.470 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:45.483+0000 7f3e56a067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:45.471 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:45.483+0000 7f3e56a067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:45.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:21:45.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:21:45.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:21:45.874 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:21:45.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:21:45.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:21:45.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:45.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:21:45.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:45.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:46.034 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:46.047+0000 7f3e56a067c0 -1 Falling back to public interface 2024-04-25T09:21:46.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:47.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:47.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:47.293 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:21:47.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:21:47.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:47.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:47.712 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:48.575 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:21:48.587+0000 7f3e56a067c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:21:48.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:48.714 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:48.715 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:21:48.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:21:48.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:48.715 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:49.221 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:50.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:50.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:50.223 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:21:50.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:21:50.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:50.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:50.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:51.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:51.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:51.690 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:21:51.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:21:51.690 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:51.691 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:52.129 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:53.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:53.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:53.132 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:21:53.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:21:53.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:53.132 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:53.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:54.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:54.636 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:54.636 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:21:54.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:21:54.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:54.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:55.033 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:56.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:56.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:56.034 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:21:56.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:21:56.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:56.035 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:56.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:57.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:57.456 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:57.457 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:21:57.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:21:57.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:57.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:57.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:58.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:58.906 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:58.906 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:21:58.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:21:58.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:58.907 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:59.427 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:22:00.429 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:22:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:00.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:22:01.265 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:02.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:22:02.268 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:22:02.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:02.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:22:02.893 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:03.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:03.894 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:03.895 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:22:03.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:22:03.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:03.895 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:22:04.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:05.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:05.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:05.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:22:05.646 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:22:05.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:05.647 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:22:06.183 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:07.184 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:07.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:07.185 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:22:07.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:22:07.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:07.185 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:22:08.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:09.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:09.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:09.109 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:22:09.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:22:09.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:09.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:22:09.944 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:10.945 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:10.946 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:22:10.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:22:10.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:10.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:22:11.370 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2987160473,v1:127.0.0.1:6803/2987160473] [v2:127.0.0.1:6804/2987160473,v1:127.0.0.1:6805/2987160473] exists,up 41754d99-56eb-45bf-8190-0ad3ef6c2fe7 2024-04-25T09:22:11.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:22:11.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:22:11.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:22:11.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:22:11.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:22:11.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:22:11.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:22:11.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:22:11.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:22:11.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:22:11.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb2ca43d-0f2e-4908-96a6-e68f0f5d7ccf 2024-04-25T09:22:11.378 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 bb2ca43d-0f2e-4908-96a6-e68f0f5d7ccf 2024-04-25T09:22:11.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bb2ca43d-0f2e-4908-96a6-e68f0f5d7ccf' 2024-04-25T09:22:11.379 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:22:11.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDICpmFuyEGBAATcau5AMS48Z2MmrAsH5rhA== 2024-04-25T09:22:11.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDICpmFuyEGBAATcau5AMS48Z2MmrAsH5rhA=="}' 2024-04-25T09:22:11.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb2ca43d-0f2e-4908-96a6-e68f0f5d7ccf -i td/repro-long-log/1/new.json 2024-04-25T09:22:12.078 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:22:12.097 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:22:12.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDICpmFuyEGBAATcau5AMS48Z2MmrAsH5rhA== --osd-uuid bb2ca43d-0f2e-4908-96a6-e68f0f5d7ccf 2024-04-25T09:22:12.132 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:12.146+0000 7f4abbf507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:12.164 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:12.178+0000 7f4abbf507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:12.165 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:12.178+0000 7f4abbf507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:12.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:12.178+0000 7f4abbf507c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:22:12.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:12.178+0000 7f4abbf507c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:22:12.166 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:12.178+0000 7f4abbf507c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:22:12.203 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:12.214+0000 7f4abbf507c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:22:18.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:22:18.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:22:18.382 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T09:22:18.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:22:18.383 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:22:18.940 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:22:18.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:22:18.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:22:18.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:22:18.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:22:18.941 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:22:18.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:18.982+0000 7fcf97d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:18.971 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:18.982+0000 7fcf97d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:18.972 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:18.986+0000 7fcf97d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:19.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:22:19.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:22:19.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:22:19.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:22:19.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:22:19.372 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:19.373 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:22:19.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:22:19.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:19.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:19.541 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:19.554+0000 7fcf97d1f7c0 -1 Falling back to public interface 2024-04-25T09:22:19.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:20.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:20.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:20.844 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:22:20.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:22:20.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:20.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:21.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:22.091 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:22.102+0000 7fcf97d1f7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:22:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:22.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:22.293 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:22:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:22:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:22.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:22.828 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:23.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:23.830 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:23.830 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:22:23.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:22:23.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:23.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:24.259 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:25.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:25.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:25.261 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:22:25.261 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:22:25.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:25.262 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:25.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:26.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:26.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:26.685 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:22:26.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:22:26.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:26.685 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:27.249 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:28.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:28.251 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:28.251 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:22:28.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:22:28.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:28.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:28.680 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:29.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:29.681 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:29.682 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:22:29.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:22:29.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:29.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:30.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:31.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:31.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:31.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:22:31.233 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:22:31.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:31.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:31.654 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:32.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:32.656 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:32.656 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:22:32.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:22:32.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:32.657 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:33.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:34.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:34.092 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:34.093 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:22:34.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:22:34.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:34.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:34.520 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:35.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:35.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:35.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:22:35.522 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:22:35.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:35.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:36.447 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:37.448 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:37.449 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:37.449 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:22:37.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:22:37.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:37.450 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:38.343 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:39.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:39.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:39.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:22:39.345 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:22:39.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:39.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:40.118 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:41.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:41.119 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:41.120 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:22:41.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:22:41.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:41.120 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:42.385 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:43.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:43.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:43.387 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:22:43.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:22:43.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:43.388 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:44.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:45.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:45.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:45.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:22:45.377 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:22:45.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:45.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:22:45.807 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/51577289,v1:127.0.0.1:6811/51577289] [v2:127.0.0.1:6812/51577289,v1:127.0.0.1:6813/51577289] exists,up bb2ca43d-0f2e-4908-96a6-e68f0f5d7ccf 2024-04-25T09:22:45.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:22:45.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:22:45.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:22:45.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:22:45.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:22:45.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:22:45.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:22:45.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:22:45.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:22:45.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:22:45.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:22:45.813 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:22:45.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b7c9e20-d611-4bf5-94a9-b1a52de0ee69 2024-04-25T09:22:45.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2b7c9e20-d611-4bf5-94a9-b1a52de0ee69' 2024-04-25T09:22:45.814 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 2b7c9e20-d611-4bf5-94a9-b1a52de0ee69 2024-04-25T09:22:45.814 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:22:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDlICpmVqWNMhAAhtcsDwWG8VkLLQafwpNQ8g== 2024-04-25T09:22:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDlICpmVqWNMhAAhtcsDwWG8VkLLQafwpNQ8g=="}' 2024-04-25T09:22:45.831 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b7c9e20-d611-4bf5-94a9-b1a52de0ee69 -i td/repro-long-log/2/new.json 2024-04-25T09:22:46.299 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:22:46.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:22:46.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlICpmVqWNMhAAhtcsDwWG8VkLLQafwpNQ8g== --osd-uuid 2b7c9e20-d611-4bf5-94a9-b1a52de0ee69 2024-04-25T09:22:46.347 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:46.362+0000 7fea9be9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:46.385 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:46.398+0000 7fea9be9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:46.387 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:46.398+0000 7fea9be9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:46.388 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:46.402+0000 7fea9be9c7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:22:46.388 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:46.402+0000 7fea9be9c7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:22:46.388 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:46.402+0000 7fea9be9c7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:22:46.426 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:46.442+0000 7fea9be9c7c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:22:52.793 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:22:52.794 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:22:52.794 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T09:22:52.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:22:52.795 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:22:53.359 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:22:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:22:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:22:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:22:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:22:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:22:53.389 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:53.406+0000 7f28574f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:53.466+0000 7f28574f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:53.466+0000 7f28574f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:22:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:22:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:22:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:22:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:22:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:53.813 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:22:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:22:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:54.022 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:54.038+0000 7f28574f97c0 -1 Falling back to public interface 2024-04-25T09:22:54.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:55.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:55.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:55.234 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:22:55.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:22:55.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:55.235 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:55.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:56.540 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:22:56.558+0000 7f28574f97c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:22:56.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:56.633 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:56.633 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:22:56.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:22:56.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:56.634 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:57.355 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:58.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:58.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:58.357 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:22:58.357 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:22:58.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:58.358 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:58.780 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:59.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:59.782 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:59.782 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:22:59.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:22:59.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:59.783 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:00.415 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:01.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:01.417 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:01.417 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:23:01.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:23:01.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:01.418 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:01.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:02.874 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:23:02.874 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:23:02.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:02.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:03.473 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:04.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:04.474 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:04.475 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:23:04.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:23:04.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:04.475 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:04.915 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:05.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:05.917 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:05.917 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:23:05.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:23:05.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:05.918 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:06.399 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:07.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:07.401 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:07.401 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:23:07.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:23:07.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:07.402 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:07.901 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:08.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:08.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:08.904 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:23:08.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:23:08.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:08.905 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:09.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:10.428 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:23:10.429 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:23:10.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:10.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:11.222 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:12.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:12.223 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:12.224 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:23:12.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:23:12.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:12.224 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:12.972 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:13.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:13.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:13.974 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:23:13.975 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:23:13.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:13.975 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:15.074 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:16.075 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:16.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:16.076 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:23:16.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:23:16.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:16.076 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:16.735 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:17.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:17.737 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:17.737 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:23:17.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:23:17.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:17.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:18.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:19.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:19.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:19.844 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:23:19.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:23:19.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:19.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:23:20.269 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3379560523,v1:127.0.0.1:6819/3379560523] [v2:127.0.0.1:6820/3379560523,v1:127.0.0.1:6821/3379560523] exists,up 2b7c9e20-d611-4bf5-94a9-b1a52de0ee69 2024-04-25T09:23:20.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:23:20.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:23:20.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:23:20.271 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:23:20.829 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T09:23:20.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:23:20.846 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:23:21.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:23:21.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:23:21.312 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:23:21.457 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:21.483 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:21.509 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:21.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:23:21.661 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:21.688 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:21.720 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:21.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:23:21.870 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:21.897 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:21.923 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:21.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:23:22.103 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:22.131 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:22.161 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:22.175 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:23:22.311 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:22.339 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:22.365 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:22.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:23:22.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:23:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:22.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:22.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:22.763 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:23.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:23.056 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:23.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:23.190 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:23.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:23.316 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:23.441 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:23.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:23.558 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:23.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:23.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:23.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:23.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:23.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:23.952 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:24.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:24.087 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:24.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:24.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:24.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:24.362 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:24.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:24.487 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:24.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:24.605 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:24.722 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:24.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:24.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:24.840 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:24.956 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:24.957 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:25.090 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:25.091 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:25.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:25.250 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:23:25.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:23:25.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:23:25.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:23:25.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:23:25.367 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:23:25.510 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836499 2024-04-25T09:23:25.534 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672978 2024-04-25T09:23:25.559 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509446 2024-04-25T09:23:25.573 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:23:28.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:28.576 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:28.737 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:23:28.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:28.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:28.738 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:23:28.890 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:23:28.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:28.891 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:29.042 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:29.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:29.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:29.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:23:29.231 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:29.231 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:23:29.362 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:23:29.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:29.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:29.507 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:29.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:29.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:29.625 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:29.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:29.628 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:29.759 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:29.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:29.762 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:29.893 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:29.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:30.077 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:30.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:30.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:30.194 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:30.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:30.198 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:30.312 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:30.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:30.315 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:30.429 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:30.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:30.433 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:30.570 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:30.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:30.574 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:30.696 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:30.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:30.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:30.847 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:30.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:30.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:30.964 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:30.968 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:30.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:31.082 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:31.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:31.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:31.215 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:31.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:31.218 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:31.332 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:31.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:31.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:31.450 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:31.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:31.453 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:31.567 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:31.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:31.570 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:31.717 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:31.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:31.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:31.885 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:31.889 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:23:31.890 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:23:32.019 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:32.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:23:32.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:23:32.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:23:32.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:23:32.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:23:32.167 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836503 2024-04-25T09:23:32.191 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672982 2024-04-25T09:23:32.216 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509449 2024-04-25T09:23:32.230 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:23:35.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:35.233 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:35.397 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T09:23:35.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:35.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:35.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:23:35.544 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T09:23:35.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:35.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:35.710 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:35.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:35.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:35.711 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:23:35.861 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:35.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:124: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-04-25T09:23:36.000 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:36.026 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:36.059 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:36.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:125: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-04-25T09:23:36.213 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:36.240 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:36.271 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:36.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:126: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-04-25T09:23:36.424 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:36.452 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:36.481 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:36.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:127: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-04-25T09:23:36.632 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:36.662 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:36.689 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:36.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:128: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 0 2024-04-25T09:23:36.830 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:23:36.858 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:23:36.886 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:23:36.899 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:131: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:23:37.042 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:37.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:132: TEST_trim_max_entries: test_log_size 1.0 18 2024-04-25T09:23:37.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:23:37.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-04-25T09:23:37.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:23:37.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:23:37.252 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836506 2024-04-25T09:23:37.281 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672985 2024-04-25T09:23:37.310 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509452 2024-04-25T09:23:37.326 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:23:40.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:40.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:40.496 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:23:40.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:40.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:40.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-04-25T09:23:40.654 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:23:40.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:40.655 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:40.815 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:40.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:40.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:40.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:23:40.970 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:40.971 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:133: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:23:41.136 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:41.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:134: TEST_trim_max_entries: test_log_size 1.0 15 2024-04-25T09:23:41.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:23:41.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-04-25T09:23:41.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:23:41.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:23:41.287 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836508 2024-04-25T09:23:41.311 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672988 2024-04-25T09:23:41.335 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509455 2024-04-25T09:23:41.350 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:23:44.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:44.352 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:44.514 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:23:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:44.515 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:44.516 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-04-25T09:23:44.673 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:23:44.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:44.674 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:44.831 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:44.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:44.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:44.832 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:23:44.980 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:44.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:135: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:23:45.105 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:45.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:136: TEST_trim_max_entries: test_log_size 1.0 12 2024-04-25T09:23:45.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:23:45.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-04-25T09:23:45.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:23:45.110 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:23:45.249 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836511 2024-04-25T09:23:45.271 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672990 2024-04-25T09:23:45.296 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509458 2024-04-25T09:23:45.310 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:23:48.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:48.313 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:48.479 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:23:48.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:48.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:48.480 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-04-25T09:23:48.636 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:23:48.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:48.637 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:48.790 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:48.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:48.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:48.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:23:48.935 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:48.936 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:137: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:23:49.057 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:49.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:138: TEST_trim_max_entries: test_log_size 1.0 9 2024-04-25T09:23:49.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:23:49.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-04-25T09:23:49.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:23:49.061 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:23:49.219 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836514 2024-04-25T09:23:49.244 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672993 2024-04-25T09:23:49.269 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509460 2024-04-25T09:23:49.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:23:52.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:52.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:52.451 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:23:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:52.452 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-04-25T09:23:52.610 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:23:52.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:52.610 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:52.764 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:52.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:52.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:52.765 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:23:52.921 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:52.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:139: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:23:53.068 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:53.071 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:140: TEST_trim_max_entries: test_log_size 1.0 6 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:23:53.222 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836517 2024-04-25T09:23:53.246 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672996 2024-04-25T09:23:53.270 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509463 2024-04-25T09:23:53.284 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:23:56.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:56.287 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:56.454 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:23:56.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:56.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:23:56.455 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-04-25T09:23:56.606 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:23:56.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:56.607 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:56.758 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:56.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:23:56.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:23:56.759 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:23:56.897 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:23:56.897 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:141: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:23:57.029 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:23:57.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:142: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:23:57.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:23:57.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:23:57.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:23:57.032 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:23:57.169 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836520 2024-04-25T09:23:57.192 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672999 2024-04-25T09:23:57.215 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509466 2024-04-25T09:23:57.229 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:24:00.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:00.232 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:00.395 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:24:00.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:00.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:00.396 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:24:00.542 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:24:00.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:00.543 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:00.696 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:00.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:00.696 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:24:00.697 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:00.838 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:00.839 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:145: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:24:00.956 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:24:00.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:146: TEST_trim_max_entries: test_log_size 1.0 4 2024-04-25T09:24:00.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:24:00.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:24:00.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:24:00.960 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:24:01.098 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836522 2024-04-25T09:24:01.122 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949673002 2024-04-25T09:24:01.147 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509469 2024-04-25T09:24:01.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:24:04.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:04.164 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:04.330 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:24:04.330 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:04.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:04.331 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:24:04.489 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:24:04.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:04.490 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:04.643 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:04.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:04.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:04.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:24:04.802 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:04.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:147: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:24:05.025 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:24:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:148: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:24:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:24:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:24:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:24:05.030 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:24:05.175 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836525 2024-04-25T09:24:05.199 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949673004 2024-04-25T09:24:05.223 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509472 2024-04-25T09:24:05.238 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:24:08.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:08.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:08.405 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:24:08.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:08.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:08.406 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:24:08.563 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:24:08.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:08.563 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:08.718 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:08.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:08.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:08.719 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:24:08.871 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:08.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:149: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:24:09.002 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:24:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:150: TEST_trim_max_entries: test_log_size 1.0 4 2024-04-25T09:24:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:24:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:24:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:24:09.006 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:24:09.147 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836528 2024-04-25T09:24:09.174 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949673007 2024-04-25T09:24:09.196 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509474 2024-04-25T09:24:09.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:24:12.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:12.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:12.378 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:24:12.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:12.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:12.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:24:12.530 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:24:12.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:12.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:12.683 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:12.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:12.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:12.684 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:24:12.832 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:12.833 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:151: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:24:12.947 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:24:12.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:152: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:24:12.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:24:12.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:24:12.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:24:12.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:24:13.090 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836531 2024-04-25T09:24:13.114 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949673010 2024-04-25T09:24:13.138 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509477 2024-04-25T09:24:13.153 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:24:16.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:16.155 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:16.318 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:24:16.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:16.319 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:24:16.320 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:24:16.478 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:24:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:16.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:16.630 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:16.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:24:16.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:24:16.631 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:24:16.787 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:24:16.788 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:24:17.099 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:24:17.099 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:24:17.100 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:24:17.101 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:24:17.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:24:17.103 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:24:17.104 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:24:17.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:24:17.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:24:17.105 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:24:17.106 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:24:17.107 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:24:17.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:24:17.109 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:24:17.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:24:17.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:17.131 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:17.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:24:17.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:24:17.133 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:24:17.134 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:24:17.136 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:24:17.136 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:24:17.137 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:24:17.138 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:24:17.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:24:17.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:24:17.139 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:24:17.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:24:17.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:24:17.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:24:17.141 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:24:17.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:24:17.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:24:17.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:24:17.144 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:24:17.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:17.145 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:17.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:24:17.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:24:17.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:24:17.146 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:24:17.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:24:17.147 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:17.148 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:17.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.493765 2024-04-25T09:24:17.150 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries_with_dups td/repro-long-log 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:157: TEST_trim_max_entries_with_dups: local dir=td/repro-long-log 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:159: TEST_trim_max_entries_with_dups: setup_log_test td/repro-long-log 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:24:17.151 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:24:17.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:24:17.617 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:24:17.618 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:24:17.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:24:17.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:17.619 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:17.619 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:24:17.619 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:24:17.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:24:17.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:24:17.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:24:17.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:24:17.847 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:24:17.848 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:24:17.848 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:24:17.848 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:24:17.848 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:24:17.849 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:24:17.849 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:17.849 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:17.849 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493765/ceph-mon.a.asok 2024-04-25T09:24:17.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:24:17.849 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493765/ceph-mon.a.asok config get fsid 2024-04-25T09:24:17.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:24:17.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:24:17.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:24:17.968 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:24:17.970 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:24:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:24:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:24:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:24:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:24:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:17.970 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493765/ceph-mon.a.asok 2024-04-25T09:24:17.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:24:17.971 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493765/ceph-mon.a.asok config get mon_host 2024-04-25T09:24:18.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:24:18.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:24:18.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:24:18.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:24:18.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:24:18.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:24:18.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:24:19.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:24:19.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:24:19.000 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:24:19.001 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:24:19.001 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:19.001 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:19.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:24:19.001 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:24:19.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:24:19.042 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:24:19.043 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=008875e5-2a01-4968-aeeb-4a2796686d9e 2024-04-25T09:24:19.044 INFO:tasks.workunit.client.0.smithi088.stdout:add osd0 008875e5-2a01-4968-aeeb-4a2796686d9e 2024-04-25T09:24:19.045 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 008875e5-2a01-4968-aeeb-4a2796686d9e' 2024-04-25T09:24:19.045 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:24:19.058 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDISpmF8OKBBAAGrburIffq4B9MQcMxuqPUA== 2024-04-25T09:24:19.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDISpmF8OKBBAAGrburIffq4B9MQcMxuqPUA=="}' 2024-04-25T09:24:19.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 008875e5-2a01-4968-aeeb-4a2796686d9e -i td/repro-long-log/0/new.json 2024-04-25T09:24:19.349 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:19.363 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:24:19.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDISpmF8OKBBAAGrburIffq4B9MQcMxuqPUA== --osd-uuid 008875e5-2a01-4968-aeeb-4a2796686d9e 2024-04-25T09:24:19.395 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:19.408+0000 7fdbec1467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:19.418 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:19.432+0000 7fdbec1467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:19.418 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:19.432+0000 7fdbec1467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:19.432+0000 7fdbec1467c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:24:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:19.432+0000 7fdbec1467c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:24:19.419 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:19.432+0000 7fdbec1467c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:24:19.457 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:19.472+0000 7fdbec1467c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:24:25.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:24:25.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:24:25.522 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd0 key to auth repository 2024-04-25T09:24:25.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:24:25.522 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:24:26.071 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.0 2024-04-25T09:24:26.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:24:26.072 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:24:26.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:24:26.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:24:26.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:24:26.101 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:26.116+0000 7f30c0b217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:26.131 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:26.148+0000 7f30c0b217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:26.133 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:26.148+0000 7f30c0b217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:26.495 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:24:26.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:24:26.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:24:26.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:24:26.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:24:26.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:26.496 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:24:26.496 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:24:26.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:26.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:26.697 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:26.712+0000 7f30c0b217c0 -1 Falling back to public interface 2024-04-25T09:24:26.922 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:27.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:27.924 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:27.924 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:24:27.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:24:27.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:27.925 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:28.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:29.213 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:29.228+0000 7f30c0b217c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:24:29.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:29.338 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:29.339 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:24:29.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:24:29.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:29.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:29.852 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:30.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:30.854 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:30.854 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:24:30.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:24:30.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:30.855 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:31.373 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:32.374 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:32.375 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:32.375 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:24:32.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:24:32.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:32.376 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:32.812 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:33.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:33.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:33.814 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:24:33.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:24:33.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:33.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:34.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:35.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:35.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:35.246 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:24:35.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:24:35.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:35.247 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:35.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:36.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:36.666 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:24:36.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:24:36.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:36.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:37.145 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:38.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:38.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:38.147 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:24:38.148 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:24:38.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:38.148 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:38.581 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:39.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:39.583 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:39.584 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:24:39.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:24:39.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:39.584 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:40.078 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:41.079 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:41.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:41.080 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:24:41.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:24:41.080 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:41.081 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:41.510 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:42.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:42.512 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:42.512 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:24:42.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:24:42.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:42.513 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:43.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:44.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:44.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:44.021 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:24:44.022 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:24:44.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:44.022 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:44.996 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:45.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:45.998 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:45.998 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:24:45.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:24:45.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:45.999 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:46.700 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:47.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:47.702 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:47.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:24:47.703 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:24:47.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:47.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:48.420 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:49.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:49.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:49.423 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:24:49.423 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:24:49.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:49.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:50.387 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:51.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:51.389 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:51.390 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:24:51.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:24:51.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:51.390 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:51.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:52.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:52.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:52.817 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:24:52.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:24:52.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:52.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:53.286 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3682577135,v1:127.0.0.1:6803/3682577135] [v2:127.0.0.1:6804/3682577135,v1:127.0.0.1:6805/3682577135] exists,up 008875e5-2a01-4968-aeeb-4a2796686d9e 2024-04-25T09:24:53.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:24:53.290 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:24:53.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:24:53.292 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:24:53.293 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:24:53.293 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=178694fa-9a3c-4d6d-b053-0e2923279559 2024-04-25T09:24:53.293 INFO:tasks.workunit.client.0.smithi088.stdout:add osd1 178694fa-9a3c-4d6d-b053-0e2923279559 2024-04-25T09:24:53.294 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 178694fa-9a3c-4d6d-b053-0e2923279559' 2024-04-25T09:24:53.294 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:24:53.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlISpmmztvExAA0CpK68rznmgirby0Qo7zfg== 2024-04-25T09:24:53.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlISpmmztvExAA0CpK68rznmgirby0Qo7zfg=="}' 2024-04-25T09:24:53.308 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 178694fa-9a3c-4d6d-b053-0e2923279559 -i td/repro-long-log/1/new.json 2024-04-25T09:24:53.774 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:24:53.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:24:53.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlISpmmztvExAA0CpK68rznmgirby0Qo7zfg== --osd-uuid 178694fa-9a3c-4d6d-b053-0e2923279559 2024-04-25T09:24:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:53.831+0000 7f6c798ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:53.831+0000 7f6c798ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:53.831+0000 7f6c798ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:53.835+0000 7f6c798ea7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:24:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:53.835+0000 7f6c798ea7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:24:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:53.835+0000 7f6c798ea7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:24:53.909 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:24:53.923+0000 7f6c798ea7c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:24:59.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:24:59.721 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:24:59.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:24:59.723 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd1 key to auth repository 2024-04-25T09:24:59.723 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:25:00.266 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.1 2024-04-25T09:25:00.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:25:00.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:25:00.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:25:00.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:25:00.267 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:25:00.295 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:00.307+0000 7f8c26e427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:00.297 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:00.311+0000 7f8c26e427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:00.298 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:00.311+0000 7f8c26e427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:00.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:25:00.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:25:00.703 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:25:00.704 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:25:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:25:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:25:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:00.704 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:25:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:00.705 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:00.841 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:00.855+0000 7f8c26e427c0 -1 Falling back to public interface 2024-04-25T09:25:01.139 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:02.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:02.140 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:02.141 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:25:02.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:25:02.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:02.141 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:02.537 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:03.406 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:03.419+0000 7f8c26e427c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:25:03.538 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:03.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:03.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:25:03.539 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:25:03.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:03.539 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:04.093 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:05.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:05.095 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:05.096 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:25:05.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:25:05.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:05.096 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:05.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:06.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:06.523 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:06.523 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:25:06.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:25:06.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:06.524 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:07.007 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:08.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:08.009 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:08.009 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:25:08.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:25:08.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:08.010 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:08.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:09.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:09.645 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:09.646 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:25:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:25:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:09.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:10.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:11.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:11.062 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:11.062 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:25:11.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:25:11.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:11.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:11.489 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:12.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:12.492 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:12.492 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:25:12.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:25:12.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:12.493 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:12.946 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:13.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:13.948 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:13.948 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:25:13.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:25:13.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:13.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:14.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:15.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:15.379 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:25:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:25:15.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:15.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:15.883 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:16.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:16.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:16.885 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:25:16.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:25:16.885 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:16.886 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:17.664 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:18.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:18.666 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:18.667 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:25:18.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:25:18.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:18.667 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:19.345 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:20.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:20.346 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:20.347 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:25:20.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:25:20.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:20.347 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:21.057 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:22.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:22.059 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:22.059 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:25:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:25:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:22.060 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:22.808 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:23.809 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:23.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:23.810 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:25:23.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:25:23.810 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:23.811 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:24.599 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:25.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:25.601 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:25.601 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:25:25.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:25:25.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:25.602 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:26.266 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:27.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:27.268 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:27.269 INFO:tasks.workunit.client.0.smithi088.stdout:17 2024-04-25T09:25:27.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:25:27.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:27.269 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:25:27.727 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3406066106,v1:127.0.0.1:6811/3406066106] [v2:127.0.0.1:6812/3406066106,v1:127.0.0.1:6813/3406066106] exists,up 178694fa-9a3c-4d6d-b053-0e2923279559 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:25:27.731 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:27.732 INFO:tasks.workunit.client.0.smithi088.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:25:27.733 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:25:27.734 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:25:27.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c0b6f85-f21d-4319-9f74-f85e77f6119c 2024-04-25T09:25:27.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4c0b6f85-f21d-4319-9f74-f85e77f6119c' 2024-04-25T09:25:27.734 INFO:tasks.workunit.client.0.smithi088.stdout:add osd2 4c0b6f85-f21d-4319-9f74-f85e77f6119c 2024-04-25T09:25:27.735 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:25:27.748 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHISpmvoWtLRAAG1U4ykAOnOX4Nhsp9BH2oQ== 2024-04-25T09:25:27.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHISpmvoWtLRAAG1U4ykAOnOX4Nhsp9BH2oQ=="}' 2024-04-25T09:25:27.749 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c0b6f85-f21d-4319-9f74-f85e77f6119c -i td/repro-long-log/2/new.json 2024-04-25T09:25:28.219 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:25:28.234 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:25:28.236 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHISpmvoWtLRAAG1U4ykAOnOX4Nhsp9BH2oQ== --osd-uuid 4c0b6f85-f21d-4319-9f74-f85e77f6119c 2024-04-25T09:25:28.267 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:28.283+0000 7f2efeb437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:28.269 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:28.283+0000 7f2efeb437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:28.270 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:28.283+0000 7f2efeb437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:28.283+0000 7f2efeb437c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:25:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:28.283+0000 7f2efeb437c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:25:28.271 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:28.283+0000 7f2efeb437c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:25:28.305 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:28.319+0000 7f2efeb437c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:25:34.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:25:34.142 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:25:34.143 INFO:tasks.workunit.client.0.smithi088.stdout:adding osd2 key to auth repository 2024-04-25T09:25:34.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:25:34.143 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:25:34.681 INFO:tasks.workunit.client.0.smithi088.stdout:start osd.2 2024-04-25T09:25:34.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:25:34.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:25:34.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:25:34.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=42159f21-ef27-4ab1-8baa-a4111a0d7db7 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493765/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:25:34.682 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:25:34.711 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:34.723+0000 7f34ec96d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:34.713 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:34.727+0000 7f34ec96d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:34.714 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:34.727+0000 7f34ec96d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:35.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:25:35.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:25:35.159 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:25:35.159 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:25:35.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:25:35.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:25:35.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:35.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:25:35.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:35.160 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:35.276 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:35.290+0000 7f34ec96d7c0 -1 Falling back to public interface 2024-04-25T09:25:35.589 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:36.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:36.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:36.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:25:36.591 INFO:tasks.workunit.client.0.smithi088.stdout:1 2024-04-25T09:25:36.591 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:36.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:36.986 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:37.871 INFO:tasks.workunit.client.0.smithi088.stderr:2024-04-25T09:25:37.882+0000 7f34ec96d7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:25:37.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:37.988 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:37.988 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:25:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:25:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:37.989 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:38.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:39.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:39.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:39.547 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:25:39.547 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:25:39.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:39.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:40.012 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:41.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:41.013 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:41.014 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:25:41.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:25:41.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:41.014 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:41.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:42.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:42.421 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:42.421 INFO:tasks.workunit.client.0.smithi088.stdout:5 2024-04-25T09:25:42.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:25:42.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:42.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:42.843 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:43.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:43.844 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:43.845 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:25:43.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:25:43.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:43.845 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:44.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:45.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:45.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:45.398 INFO:tasks.workunit.client.0.smithi088.stdout:7 2024-04-25T09:25:45.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:25:45.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:45.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:45.979 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:46.980 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:46.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:46.981 INFO:tasks.workunit.client.0.smithi088.stdout:8 2024-04-25T09:25:46.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:25:46.981 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:46.982 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:47.434 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:48.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:48.436 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:48.436 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:25:48.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:25:48.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:48.437 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:48.870 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:49.871 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:49.872 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:49.872 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:25:49.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:25:49.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:49.873 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:50.422 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:51.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:51.424 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:51.424 INFO:tasks.workunit.client.0.smithi088.stdout:11 2024-04-25T09:25:51.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:25:51.425 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:51.426 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:52.476 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:53.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:53.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:53.478 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:25:53.478 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:25:53.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:53.479 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:54.815 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:55.816 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:55.817 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:55.817 INFO:tasks.workunit.client.0.smithi088.stdout:13 2024-04-25T09:25:55.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:25:55.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:55.818 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:56.803 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:57.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:57.805 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:57.806 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:25:57.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:25:57.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:57.806 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:25:58.686 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:59.687 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:59.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:59.688 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:25:59.688 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:25:59.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:59.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:00.395 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:01.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:01.397 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:01.397 INFO:tasks.workunit.client.0.smithi088.stdout:16 2024-04-25T09:26:01.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:26:01.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:01.398 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:26:01.862 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4200782400,v1:127.0.0.1:6819/4200782400] [v2:127.0.0.1:6820/4200782400,v1:127.0.0.1:6821/4200782400] exists,up 4c0b6f85-f21d-4319-9f74-f85e77f6119c 2024-04-25T09:26:01.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:26:01.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:26:01.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:26:01.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:26:02.592 INFO:tasks.workunit.client.0.smithi088.stderr:pool 'test' already exists 2024-04-25T09:26:02.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:26:02.612 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:26:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:26:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:26:03.286 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:26:03.423 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:03.450 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:03.481 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:03.494 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:26:03.632 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:03.659 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:03.685 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:03.698 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:26:03.840 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:03.868 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:03.895 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:03.909 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:26:04.043 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:04.070 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:04.097 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:04.111 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:26:04.304 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:04.332 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:04.363 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:04.378 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:26:04.380 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:26:04.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:04.382 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:04.649 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:04.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:04.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:04.920 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:05.121 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:05.122 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:05.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:05.246 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:05.380 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:05.381 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:05.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:05.506 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:05.639 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:05.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:05.790 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:05.791 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:05.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:05.933 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:06.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:06.083 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:06.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:06.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:06.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:06.377 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:06.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:06.545 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:06.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:06.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:06.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:06.779 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:06.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:06.904 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:07.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:07.046 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:07.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:07.181 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:07.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:07.305 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:26:07.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:26:07.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:07.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:26:07.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:26:07.431 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:07.576 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836499 2024-04-25T09:26:07.600 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672974 2024-04-25T09:26:07.626 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509450 2024-04-25T09:26:07.640 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:10.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:10.644 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:10.806 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:10.807 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:26:10.961 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:10.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:10.961 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:11.116 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:26:11.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:11.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:11.117 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:26:11.272 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:26:11.272 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:26:11.417 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:26:11.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:11.419 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:11.571 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:11.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:11.575 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:11.731 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:11.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:11.734 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:11.848 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:11.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:11.851 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:11.965 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:11.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:11.969 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:12.099 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:12.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:12.102 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:12.249 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:12.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:12.252 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:12.376 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:12.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:12.379 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:12.517 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:12.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:12.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:12.643 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:12.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:12.646 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:12.785 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:12.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:12.789 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:12.944 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:12.949 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:12.950 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:13.069 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:13.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:13.073 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:13.195 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:13.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:13.199 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:13.337 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:13.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:13.340 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:13.463 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:13.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:13.466 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:13.588 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:13.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:13.592 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:13.742 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:13.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:13.745 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:13.881 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:13.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:13.884 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:13.998 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:26:14.002 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:26:14.141 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:14.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:26:14.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:14.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:26:14.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:26:14.144 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:14.290 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836503 2024-04-25T09:26:14.313 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672977 2024-04-25T09:26:14.338 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509453 2024-04-25T09:26:14.353 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:17.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:17.356 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:17.520 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T09:26:17.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:17.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:17.521 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:26:17.676 INFO:tasks.workunit.client.0.smithi088.stdout:21 2024-04-25T09:26:17.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:17.677 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:17.833 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:26:17.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:17.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:17.834 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:26:17.987 INFO:tasks.workunit.client.0.smithi088.stdout:0 2024-04-25T09:26:17.987 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:161: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-04-25T09:26:18.121 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:18.148 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:18.181 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:18.196 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:162: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-04-25T09:26:18.336 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:18.368 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:18.396 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:18.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:163: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-04-25T09:26:18.552 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:18.581 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:18.612 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:18.627 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:164: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-04-25T09:26:18.764 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:18.791 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:18.821 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:18.836 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:165: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:26:18.977 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: {} 2024-04-25T09:26:19.004 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: {} 2024-04-25T09:26:19.034 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: {} 2024-04-25T09:26:19.049 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:169: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:19.221 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:19.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:170: TEST_trim_max_entries_with_dups: test_log_size 1.0 18 2 2024-04-25T09:26:19.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:19.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-04-25T09:26:19.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=2 2024-04-25T09:26:19.226 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:19.369 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836506 2024-04-25T09:26:19.393 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672980 2024-04-25T09:26:19.417 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509456 2024-04-25T09:26:19.432 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:22.435 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:22.599 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:26:22.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:22.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:22.600 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-04-25T09:26:22.754 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:26:22.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:22.755 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:22.909 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:26:22.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:22.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:22.910 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 2 2024-04-25T09:26:23.062 INFO:tasks.workunit.client.0.smithi088.stdout:2 2024-04-25T09:26:23.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:171: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:23.198 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:23.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:172: TEST_trim_max_entries_with_dups: test_log_size 1.0 15 6 2024-04-25T09:26:23.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:23.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-04-25T09:26:23.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=6 2024-04-25T09:26:23.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:23.346 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836509 2024-04-25T09:26:23.369 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672983 2024-04-25T09:26:23.399 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509459 2024-04-25T09:26:23.410 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:26.412 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:26.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:26.577 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:26:26.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:26.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:26.578 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-04-25T09:26:26.725 INFO:tasks.workunit.client.0.smithi088.stdout:15 2024-04-25T09:26:26.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:26.725 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:26.875 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:26:26.875 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:26.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:26.876 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 6 2024-04-25T09:26:27.022 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:26:27.023 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:173: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:27.158 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:27.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:174: TEST_trim_max_entries_with_dups: test_log_size 1.0 12 10 2024-04-25T09:26:27.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:27.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-04-25T09:26:27.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=10 2024-04-25T09:26:27.162 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:27.303 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836511 2024-04-25T09:26:27.327 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672986 2024-04-25T09:26:27.350 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509462 2024-04-25T09:26:27.364 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:30.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:30.368 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:30.530 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:26:30.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:30.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:30.531 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-04-25T09:26:30.689 INFO:tasks.workunit.client.0.smithi088.stdout:12 2024-04-25T09:26:30.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:30.689 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:30.841 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:26:30.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:30.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:30.842 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 10 2024-04-25T09:26:31.001 INFO:tasks.workunit.client.0.smithi088.stdout:10 2024-04-25T09:26:31.001 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:175: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:31.127 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:31.130 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:176: TEST_trim_max_entries_with_dups: test_log_size 1.0 9 14 2024-04-25T09:26:31.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:31.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-04-25T09:26:31.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=14 2024-04-25T09:26:31.131 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:31.273 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836514 2024-04-25T09:26:31.298 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672988 2024-04-25T09:26:31.322 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509464 2024-04-25T09:26:31.336 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:34.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:34.339 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:34.501 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:26:34.501 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:34.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-04-25T09:26:34.502 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:34.660 INFO:tasks.workunit.client.0.smithi088.stdout:9 2024-04-25T09:26:34.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:34.661 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:34.813 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:26:34.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:34.813 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:34.814 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 14 2024-04-25T09:26:34.964 INFO:tasks.workunit.client.0.smithi088.stdout:14 2024-04-25T09:26:34.964 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:177: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:35.121 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:35.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:178: TEST_trim_max_entries_with_dups: test_log_size 1.0 6 18 2024-04-25T09:26:35.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:35.125 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-04-25T09:26:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-04-25T09:26:35.126 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:35.266 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836517 2024-04-25T09:26:35.290 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672991 2024-04-25T09:26:35.314 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509467 2024-04-25T09:26:35.329 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:38.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:38.332 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:38.497 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:26:38.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:38.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:38.497 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-04-25T09:26:38.649 INFO:tasks.workunit.client.0.smithi088.stdout:6 2024-04-25T09:26:38.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:38.650 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:38.801 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:26:38.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:38.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:38.802 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-04-25T09:26:38.954 INFO:tasks.workunit.client.0.smithi088.stdout:18 2024-04-25T09:26:38.954 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:179: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:39.081 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:39.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:180: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:26:39.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:39.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:26:39.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:26:39.085 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:39.227 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836520 2024-04-25T09:26:39.250 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672994 2024-04-25T09:26:39.274 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509470 2024-04-25T09:26:39.289 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:42.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:42.291 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:42.456 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:26:42.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:42.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:42.457 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:26:42.616 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:26:42.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:42.616 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:42.769 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:42.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:42.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:42.770 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:26:42.926 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:42.927 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:183: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:43.059 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:43.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:184: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-04-25T09:26:43.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:43.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:26:43.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:26:43.063 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:43.202 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836523 2024-04-25T09:26:43.224 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949672997 2024-04-25T09:26:43.247 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509473 2024-04-25T09:26:43.260 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:46.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:46.263 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:46.429 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:26:46.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:46.429 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:46.430 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:26:46.586 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:26:46.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:46.587 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:46.740 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:46.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:46.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:46.741 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:26:46.886 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:46.887 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:185: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:47.036 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:47.039 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:186: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:26:47.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:47.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:26:47.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:26:47.040 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:47.178 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836525 2024-04-25T09:26:47.203 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949673000 2024-04-25T09:26:47.226 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509476 2024-04-25T09:26:47.241 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:50.243 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:50.244 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:50.412 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:26:50.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:50.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:50.413 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:26:50.571 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:26:50.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:50.571 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:50.729 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:50.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:50.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:50.730 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:26:50.881 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:50.882 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:187: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:51.030 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:51.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:188: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-04-25T09:26:51.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:51.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:26:51.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:26:51.034 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:51.177 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836528 2024-04-25T09:26:51.201 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949673002 2024-04-25T09:26:51.225 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509478 2024-04-25T09:26:51.239 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:54.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:54.242 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:54.406 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:26:54.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:54.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:54.407 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:26:54.567 INFO:tasks.workunit.client.0.smithi088.stdout:4 2024-04-25T09:26:54.567 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:54.568 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:54.728 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:54.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:54.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:54.729 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:26:54.881 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:54.881 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:189: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:26:55.016 INFO:tasks.workunit.client.0.smithi088.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:26:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:190: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:26:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:26:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:26:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:26:55.019 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:26:55.162 INFO:tasks.workunit.client.0.smithi088.stdout:osd.0: 21474836531 2024-04-25T09:26:55.186 INFO:tasks.workunit.client.0.smithi088.stdout:osd.1: 42949673005 2024-04-25T09:26:55.210 INFO:tasks.workunit.client.0.smithi088.stdout:osd.2: 64424509481 2024-04-25T09:26:55.225 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:26:58.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:58.228 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:58.393 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:26:58.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:58.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:26:58.394 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:26:58.547 INFO:tasks.workunit.client.0.smithi088.stdout:3 2024-04-25T09:26:58.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:58.548 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:58.708 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:58.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:26:58.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:26:58.708 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:26:58.861 INFO:tasks.workunit.client.0.smithi088.stdout:20 2024-04-25T09:26:58.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:26:58.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:26:58.862 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:26:58.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:26:58.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:26:58.863 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:26:58.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:26:58.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:26:58.863 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:26:59.172 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:26:59.172 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:26:59.174 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:26:59.174 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:26:59.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:26:59.176 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:26:59.176 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:26:59.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:59.177 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:26:59.178 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:26:59.178 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:59.179 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:26:59.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:26:59.180 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:26:59.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:26:59.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:59.202 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:26:59.202 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:26:59.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:26:59.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:26:59.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:26:59.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/repro-long-log 0 2024-04-25T09:26:59.204 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:26:59.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:26:59.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:26:59.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:26:59.205 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:26:59.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:26:59.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:26:59.205 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:26:59.207 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:26:59.207 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:26:59.208 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:26:59.208 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:26:59.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:26:59.210 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:26:59.210 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:26:59.211 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:59.213 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:26:59.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:26:59.214 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:59.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:26:59.214 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:26:59.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:26:59.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:26:59.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:59.215 INFO:tasks.workunit.client.0.smithi088.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493765 2024-04-25T09:26:59.215 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493765 2024-04-25T09:26:59.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:26:59.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:26:59.216 INFO:tasks.workunit.client.0.smithi088.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:26:59.218 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:26:59.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:26:59.231 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-04-25T09:26:59.231 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T09:26:59.591 DEBUG:teuthology.parallel:result is None 2024-04-25T09:26:59.592 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:26:59.603 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:26:59.603 DEBUG:teuthology.orchestra.run.smithi088:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T09:26:59.670 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:26:59.671 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T09:26:59.684 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T09:26:59.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T09:26:59.754 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T09:26:59.763 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T09:26:59.763 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T09:26:59.786 INFO:teuthology.orchestra.run.smithi088.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T09:26:59.787 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================== 2024-04-25T09:26:59.787 INFO:teuthology.orchestra.run.smithi088.stdout:+hv01.front.sepi 44.4.53.2 2 u 96 512 377 0.165 -3.602 0.792 2024-04-25T09:26:59.787 INFO:teuthology.orchestra.run.smithi088.stdout:+hv02.front.sepi 63.231.80.2 3 u 429 512 377 0.160 -0.520 0.837 2024-04-25T09:26:59.787 INFO:teuthology.orchestra.run.smithi088.stdout:*hv03.front.sepi 74.6.168.72 3 u 439 512 377 0.205 -1.912 1.004 2024-04-25T09:26:59.787 INFO:teuthology.orchestra.run.smithi088.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T09:26:59.788 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T09:26:59.797 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T09:26:59.797 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T09:26:59.806 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T09:26:59.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T09:26:59.825 INFO:teuthology.task.internal:Duration was 9998.674428 seconds 2024-04-25T09:26:59.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T09:26:59.834 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T09:26:59.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T09:26:59.870 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T09:26:59.871 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-04-25T09:26:59.871 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T09:26:59.924 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T09:26:59.924 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T09:27:00.000 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T09:27:00.000 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T09:27:00.105 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T09:27:00.114 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T09:27:00.115 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T09:27:00.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T09:27:00.135 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:27:00.177 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-04-25T09:27:00.188 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:27:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:27:00.237 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T09:27:00.247 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T09:27:00.249 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672895/remote/smithi088 2024-04-25T09:27:00.250 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T09:27:00.299 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T09:27:00.299 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T09:27:00.343 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T09:27:00.354 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T09:27:00.354 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T09:27:00.363 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T09:27:00.363 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T09:27:00.391 INFO:teuthology.orchestra.run.smithi088.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 09:27 /home/ubuntu/cephtest 2024-04-25T09:27:00.393 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T09:27:00.403 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T09:27:00.426 INFO:teuthology.run:Summary data: description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 9998.674427509308 flavor: default owner: scheduled_nmordech@teuthology success: true 2024-04-25T09:27:00.426 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T09:27:00.506 INFO:teuthology.run:pass